Agenda
- Recapping last week
- Discussing the assumptions of LMEs
- Should variable be fixed or random effects?
- Some pitfalls.
library(tidyverse)
library(broom)
library(ggplot2)
library(lme4)
library(multivariate2017)
ayT <- ay %>%
filter(plt_vclass == "ay0") %>%
mutate(dob = year-age,
decade = (dob - 1900)/10,
log2dur = log2(dur),
dur_c = log2dur - median(log2dur))
Recapping LME
Let’s recap what we learned about about mixed effect models last week. First, it is a reasonable thing to want to know that the relationship between duration and the F1 of /ay/ is:
ayT %>%
ggplot(aes(dur_c, F1_n))+
geom_point()+
stat_smooth(method = 'lm')+
theme_bw()
But if we fit just a flat model, there are many different violations of statistical assumptions, and the model is overconfident.
ayT_flat_model <- lm(F1_n ~ dur_c, data = ayT)
summary(ayT_flat_model)
Call:
lm(formula = F1_n ~ dur_c, data = ayT)
Residuals:
Min 1Q Median 3Q Max
-2.4731 -0.3900 -0.0324 0.3498 5.2217
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.487737 0.004161 117.22 <2e-16 ***
dur_c 0.435710 0.007068 61.65 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.578 on 19340 degrees of freedom
Multiple R-squared: 0.1642, Adjusted R-squared: 0.1642
F-statistic: 3800 on 1 and 19340 DF, p-value: < 2.2e-16
There is a lot of structure to the data that is being overlooked in estimating the relationship between duration and F1. First of all, it doesn’t look like every speaker has the same relationship.
random_speakers <- sample(unique(ayT$idstring), 8)
ayT%>%
filter(idstring %in% random_speakers)%>%
ggplot(aes(dur_c, F1_n))+
geom_point()+
stat_smooth(method = lm)+
facet_wrap(~idstring, ncol= 4)+
theme_bw()
Second, the data is far from balanced across speakers.
ayT %>%
count(idstring)%>%
ggplot(aes(n))+
stat_bin()+
scale_x_log10()+
xlab("n (log scale)")+
theme_bw()
Let’s look again at our estimates from the flat model:
summary(ayT_flat_model)$coef
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.4877373 0.004160927 117.21842 0
dur_c 0.4357102 0.007067868 61.64662 0
Those estimates are going to largely be a characterization of the relationship between duration and F1 for the speakers with the most data. One way, you could try to get around this problem would be to fit one linear model for every speaker.
ayT%>%
ggplot(aes(dur_c, F1_n))+
stat_smooth(method = lm,
aes(group = idstring),
se = F,
geom = "line",
alpha = 0.4)+
theme_bw()
But this isn’t ideal for a few reasons.
- We don’t wind up with a precise estimate of the general effect of duration on F1.
- We don’t wind up with an estimate of our uncertainty of the effect of duration on F1.
- None of the estimates of these models takes into account the estimates of any other model.
So, instead, we fit a mixed effects model with a random intercept and a random slope by speaker.
ayT_lme <- lmer(F1_n ~ dur_c + (1 + dur_c | idstring), data = ayT)
summary(ayT_lme)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ dur_c + (1 + dur_c | idstring)
Data: ayT
REML criterion at convergence: 27306
Scaled residuals:
Min 1Q Median 3Q Max
-5.6543 -0.6010 -0.0430 0.5386 10.3140
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 0.10060 0.3172
dur_c 0.01769 0.1330 -0.12
Residual 0.22564 0.4750
Number of obs: 19342, groups: idstring, 326
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.58113 0.01827 31.81
dur_c 0.34284 0.01071 32.00
Correlation of Fixed Effects:
(Intr)
dur_c -0.092
Here, we have the fixed effects estimates:
summary(ayT_lme)$coef
Estimate Std. Error t value
(Intercept) 0.5811306 0.0182693 31.80913
dur_c 0.3428392 0.0107126 32.00335
This is the best estimate across speakers for the relationship between F1 and duration. But we also get some valuable info about how variable around these estimates we can expect observed speakers to be!
summary(ayT_lme)$varcor
Groups Name Std.Dev. Corr
idstring (Intercept) 0.31718
dur_c 0.13301 -0.120
Residual 0.47502
The standard deviation of the random slopes here is fairly respectable. Let’s say there was a speaker who had double the slope of the group estimate. How many standard deviations would they be from the group average?
slope <- 0.3428392
slope_sd <- 0.13301
((slope * 2) - slope)/slope_sd
[1] 2.577545
Two and a half sds is fairly large, but not an extreme outlier.
The assumptions of Mixed Effects Models
Mixed effects models relax some assumptions of fixed-effects models, namely that all observations are i.i.d. It does so by allowing for partial pooling of the data according to grouping factors. However, mixed effects models are not without assumptions.
Specifically, the model assumes that the random effects are drawn from a normal distribution. The clue that the model is making this assumption is in that it gives us estimates of standard deviation of the random effects, which is a property of normal disributuons.
This assumption has a good effect when it comes to predicting the slopes and intercepts for individual speakers. Last week, we looked at the results of fitting one linear model for each speaker. Here’s the density plots of the distribution of all of the slopes and intercepts from all of those models.
Now, let’s compare these by-speaker intercepts and slopes (in grey) to the random effects from our linear mixed effects model (in yellow).
You can see that for both the intercept and the slope, but especially the slope, the range of interspeaker values has been sucked in. This is also known as “shrinkage.” If we were interested in how different the effect of duration could be between speakers, this would suggest that simply fitting a model for every speaker would overestimate the variance!
Shrinkage is desireable for a few reasons. The first is that it is often the case that extreme outlier subjects will be less extreme on a second testing (i.e. regression to the mean). So, it makes sense to moderate the estimates of outlier speakers. The second is that we ought to have higher uncertertainty for many of the speakers who are estimated to have an extreme duration effect than for those speakers whose slopes are closer to the group average
To demonstrate this point, here are the six speakers who had the biggest difference between the Maximum Likelihood estimate of their slopes (that is, from fitting a model for each speaker) and their predicted slopes from the random effects. The Maximum Likelihood estimate is the dashed black line, and the random effects predictions are in the red lines.
Compare this to the speakers whose ML estimates are the most similar to the random effects predictions.
The assumption that the random effects are drawn from a normal distribution are great because
- Shrinkage is good.
- It makes their estimation tractible on most laptops!
But as with all statistical assumptions, some caution should be taken where they may be violated. The most major, and likely most common, violation of this normality assumption is if all of the levels of your grouping factor aren’t drawn from a single distribution! For example, here is some simulated data from 6 speakers. Speakers a, b and c come from a group disribution with mean 0, and speakers d, e and f come from a group distribution with mean 4.
speakers <- letters[1:6]
speaker_means <- c(rnorm(3, mean = 0, sd = 1),
rnorm(3, mean = 4, sd = 1))
speaker_samps <- map(speaker_means, ~rnorm(50, mean = .x, sd = 1)) %>% simplify()
fake_speakers <- data.frame(speaker = rep(speakers, each = 50),
group = rep(c("x", "y"), each = (50 * 3)),
outcome = speaker_samps)
ggplot(fake_speakers, aes(outcome, fill = group))+
stat_bin(position = "identity", alpha = 0.8)+
scale_fill_colorblind()+
theme_bw()
NA
We could fit an mixed effects model with just a random intercept by speaker.
fake_lme_1 <- lmer(outcome ~ (1|speaker), data = fake_speakers)
summary(fake_lme_1)
Linear mixed model fit by REML ['lmerMod']
Formula: outcome ~ (1 | speaker)
Data: fake_speakers
REML criterion at convergence: 878.1
Scaled residuals:
Min 1Q Median 3Q Max
-2.80366 -0.69879 0.00362 0.59143 2.88948
Random effects:
Groups Name Variance Std.Dev.
speaker (Intercept) 11.5513 3.3987
Residual 0.9735 0.9867
Number of obs: 300, groups: speaker, 6
Fixed effects:
Estimate Std. Error t value
(Intercept) 2.574 1.389 1.853
Question
What’s the problem?
The LME is estimating just 1 variance parameter for all of the speakers, thus assuming all speakers are drawn from 1 distribution, but there are actually two distributions of speakers here!
There is a fairly straightforwardw way to address this violation of the LME assumption, and that’s to include the relevant grouping predictor in the model!
fake_lme_2 <- lmer(outcome ~ group + (1|speaker),
data = fake_speakers)
summary(fake_lme_2)
Linear mixed model fit by REML ['lmerMod']
Formula: outcome ~ group + (1 | speaker)
Data: fake_speakers
REML criterion at convergence: 865.2
Scaled residuals:
Min 1Q Median 3Q Max
-2.79273 -0.69862 0.00801 0.59735 2.86336
Random effects:
Groups Name Variance Std.Dev.
speaker (Intercept) 1.5118 1.2295
Residual 0.9735 0.9867
Number of obs: 300, groups: speaker, 6
Fixed effects:
Estimate Std. Error t value
(Intercept) -0.3626 0.7144 -0.508
groupy 5.8725 1.0104 5.812
Correlation of Fixed Effects:
(Intr)
groupy -0.707
Now that we have the relevant between-speaker predictor included in the model, notice how much the variance of the random-intercepts has decreased.
This brings up an important point - If there is a crucial between-speakers, between-words, between-subject, between-items, between-whatever predictor, include it in the model.
Returning to the real data example of ayT, there is one one major interspeaker variable which was not included in the model – date of birth. Plotting F1 against both decade and duration, there seem to be reliable effects. But because date of birth is a between-speakers predictor, when it wasn’t included in the model, this between speaker variation was accounted for by by-speaker random intercepts.
ayT_lme_multi <- lmer(F1_n ~ decade * dur_c + (1 + dur_c | idstring),
data = ayT)
summary(ayT_lme_multi)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ decade * dur_c + (1 + dur_c | idstring)
Data: ayT
REML criterion at convergence: 26966.3
Scaled residuals:
Min 1Q Median 3Q Max
-5.6483 -0.6007 -0.0478 0.5410 10.2455
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 0.02879 0.1697
dur_c 0.01646 0.1283 0.19
Residual 0.22576 0.4751
Number of obs: 19342, groups: idstring, 326
Fixed effects:
Estimate Std. Error t value
(Intercept) 1.099488 0.023338 47.11
decade -0.116161 0.004603 -25.23
dur_c 0.267381 0.023780 11.24
decade:dur_c 0.015591 0.004517 3.45
Correlation of Fixed Effects:
(Intr) decade dur_c
decade -0.892
dur_c 0.032 -0.037
decade:dr_c -0.040 0.073 -0.898
Violations of assumptions that can’t be accomodated
There are a few circumstances that might (definitely) arise in your actual data that violate the assumptions of mixed effects models that we can’t do anything about within the framework.
Different group variances
First, let’s revisit why adding the between-speaker grouping predictor was effective in satisfying the normality assumption.
fake_lme_2
Linear mixed model fit by REML ['lmerMod']
Formula: outcome ~ group + (1 | speaker)
Data: fake_speakers
REML criterion at convergence: 865.1792
Random effects:
Groups Name Std.Dev.
speaker (Intercept) 1.2295
Residual 0.9867
Number of obs: 300, groups: speaker, 6
Fixed Effects:
(Intercept) groupy
-0.3626 5.8725
The way to think about the random intercepts is that after taking into account all of the fixed effects, they’re how much remains to be explained by the pooling variables. We can visualize how the random effects structure “sees” the data by simply subtracting out the fixed effect of group y from their outcome data.
fake_speakers$outcome2 <- fake_speakers$outcome
fake_speakers$outcome2[fake_speakers$group == "y"] <- fake_speakers$outcome[fake_speakers$group == "y"] - fixef(fake_lme_2)[2]
ggplot(fake_speakers, aes(outcome2))+
stat_bin(aes(fill = group), position = "identity", alpha= 0.6)+
scale_fill_colorblind()+
theme_bw()
By subtracting out the effect of group y, we align the distribution of data from groups x and y to be on top of eachother, and then the random effects are estimated as if this were just one distribution.
But, two groups of speakers are able to differ in more than just their means! for example, it’s possible that these two groups of speakers differ not only in ther means, but also in their variances.
speakers <- letters[1:6]
speaker_means_2 <- c(rnorm(3, mean = 0, sd = 1),
rnorm(3, mean = 4, sd = 3))
# Here, speaker group 2 is sampled from a distribution with sd = 3
# instead of sd = 1, like it was in the first example.
speaker_samps_2 <- map(speaker_means_2, ~rnorm(50, mean = .x, sd = 1)) %>% simplify()
You can fit an LME to this data, but the fact that group y has a much larger variance than group x won’t be accounted for. The variance estimates for the random intercepts will wind up splitting the difference between the actual variances of groups x and y.
summary(lmer(outcome ~ group + (1|speaker), data = fake_speakers_2))
Linear mixed model fit by REML ['lmerMod']
Formula: outcome ~ group + (1 | speaker)
Data: fake_speakers_2
REML criterion at convergence: 893.4
Scaled residuals:
Min 1Q Median 3Q Max
-2.58673 -0.64736 0.00081 0.65741 2.90522
Random effects:
Groups Name Variance Std.Dev.
speaker (Intercept) 9.210 3.035
Residual 1.046 1.023
Number of obs: 300, groups: speaker, 6
Fixed effects:
Estimate Std. Error t value
(Intercept) -1.019 1.754 -0.581
groupy 6.668 2.481 2.688
Correlation of Fixed Effects:
(Intr)
groupy -0.707
Different variances within grouping factors
Let’s fit a random intercepts only model to the ayT data.
summary(ayT_intercept_mod)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ (1 | idstring)
Data: ayT
REML criterion at convergence: 30275.8
Scaled residuals:
Min 1Q Median 3Q Max
-5.1576 -0.6098 -0.0522 0.5510 9.9403
Random effects:
Groups Name Variance Std.Dev.
idstring (Intercept) 0.1235 0.3515
Residual 0.2662 0.5160
Number of obs: 19342, groups: idstring, 326
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.58630 0.02016 29.08
Let’s think about how this model is modelling the data. First, it’s assuming that the speaker random intercepts are being drawn from a normal distribution. The mean of that distribution is the fixed effects intercept estimate, 0.59. The standard deviation of this distribution is estimated to be 0.35. So, we could treat individual speakers’ means as being drawn from a normal distribution with \(\mu = 0.59\) and \(\sigma=0.35\).
But, every speaker contributed multiple data points to the entire data set. In addition to a \(\mu\), every speaker must also have a \(\sigma\). Does the LME estimate a \(\sigma\) for each speaker?
In fact, the standard deviation of the residuals could be thought of as the \(\sigma\) for each speaker. So, no, the LME does not estimate a \(\sigma\) for every speaker. Rather, it estimates 1 \(\sigma\) for all speakers.
Now, strictly speaking, this assumption that every speaker will have the same \(\sigma\) is almost certainly not going to be true! Here’s what it looks like if we estimate the standard deviation for every speaker with a reasonable amount of data (to make sure we’re getting fairly reliable estimates), compared to the single standard deviation of the residuals from the LME.
ayT %>%
group_by(idstring) %>%
filter(n() > 40) %>%
summarise(mean_F1 = mean(F1_n),
sd_F1 = sd(F1_n),
n = n()) %>%
ggplot(aes(mean_F1, sd_F1))+
geom_hline(yintercept=0.5160,
color = "red",
size = 3)+
geom_point(aes(size = n),
alpha = 0.6)+
scale_size_area()+
theme_bw()
It could be of some interest to capture the fact that not all speakers have the same \(\sigma\), and it would definitely improve the fit of the model, but this just isn’t something that can be capture with LMEs.
Non-normal data
Just like with ordinary linear regression, if your outcome variable is non-normal, it’s technically going to violate the assumptons of normality. For example, if instead of fitting a model where we tried to estimate how the pronunciation of ayT changed, we tried to fit a model to see whether the duration of /ayT/ has changed over time?
summary(lmer(dur_c ~ decade + (1|idstring), data = ayT))
Linear mixed model fit by REML ['lmerMod']
Formula: dur_c ~ decade + (1 | idstring)
Data: ayT
REML criterion at convergence: 32006.4
Scaled residuals:
Min 1Q Median 3Q Max
-2.6716 -0.7235 -0.0428 0.6669 5.3633
Random effects:
Groups Name Variance Std.Dev.
idstring (Intercept) 0.03914 0.1978
Residual 0.29649 0.5445
Number of obs: 19342, groups: idstring, 326
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.222092 0.026797 8.288
decade -0.044471 0.005299 -8.393
Correlation of Fixed Effects:
(Intr)
decade -0.891
Here, we get a fairly reliable effect (looks like /ayT/ has gotten shorter over time), but recall that the distribution of duration looks like this!
It’s not likely that either the by-speaker random intercepts nor the residuals will have a normal distribution. This is actually one scenario you can get around because it’s principled enough to log-scale the duration outcome, but there won’t always be a principled transform of skewed data like this.
Fixed Effect or Random Effect?
Before even getting to how to specify a random effects structure, people often also wonder which predictors should be included as a fixed effect, and which should be included as a random effect. For example, when looking at the relationship between duration and F1 for /ayT/, let’s suppose we had the following suspicion:
I think that different speakers might have different slopes for the relationship between duration and F1.
There are two different ways we could approach this
- Fit a linear model with an interaction between duration and speaker.
- Fit a linear mixed effects model with a random slope of duration by speaker.
Let’s just look at the first option briefly:
ayT_lm_interaction <- lm(F1_n ~ dur_c * idstring, data = ayT)
summary(ayT_lm_interaction)
I’ve actually left the summary of this model out of the notes, because it’s too long to be of any use. What we wind up with is an intercept and slope estimate for some reference speaker, and then the difference of everyother speakers’ intercepts and slopes from that reference speaker. The weirdness of choosing a reference level speaker can be ameliorted by using sum-contrasts, but that doesn’t address a more serious issue: This approach will wind up giving us estimates identical to the no-pooling method.
Each speaker’s coefficient and slope interaction is estimated without respect to the estimates of any other speaker. And that comes with all of the same shortcomings we dealt with in the no-pooling model.
On the other hand, let’s return to the broader ay data set. Recall we’ve been looking at /ay/ in strictly pre-voiceless context. But in the pre-voiced context, the relationship between F1 and the speakers’ date of birth looks alot different.
Looking at the graph, we might say:
I suspect that the relationship between F1 and decade will differ between pre-voiced and pre-voiceless contexts.
The way to model this would be to either:
- Fit a model with fixed effects interaction betweend decade and voicing.
- Fit a model with random slope of decade by voicing.
Just for illustration, let’s look at a model of this second approach:
summary(ay_lmer_slope)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ decade + (1 | idstring) + (1 + decade | plt_vclass)
Data: ay
REML criterion at convergence: 169638.8
Scaled residuals:
Min 1Q Median 3Q Max
-5.5863 -0.6114 -0.0119 0.5965 9.9651
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 0.01120 0.10585
plt_vclass (Intercept) 0.01164 0.10788
decade 0.00608 0.07797 1.00
Residual 0.43052 0.65614
Number of obs: 84713, groups: idstring, 326; plt_vclass, 2
Fixed effects:
Estimate Std. Error t value
(Intercept) 1.25160 0.07760 16.13
decade -0.07838 0.05521 -1.42
Correlation of Fixed Effects:
(Intr)
decade 0.973
We get out one fixed effects intercept and slope, and an estimate of the variance of the random slopes by voicing context. We can also look at the random effects predictions:
ranef(ay_lmer_slope)$plt_vclass
One thing that isn’t so great about this result is that shrinkage will ahve applied to these random slopes! That is, the predicted slopes for pre-voiced and pre-voiceless /ay/ are going to be less different here than they would be as fixed effects.
We can sum up the motivation for including a factor as a fixed or random effect based on whether or not we want shrinkage, or partial pooling, to apply between levels of the factor. Or, another way of putting it is whether or not we want to treat the parameter values for each level of the factor as being drawn from a population-level distribution. That is, the decision is always going to be guided by your conceptualization of the data, even though there are usually decisions that can be made by convention. Almost always, the following can be treated as random effects
- individuals (speakers, subjects, texts, authors, etc.)
- items (words, specific stimuli, etc.)
And most things that you want to look at for a generalizable effect can be fixed effects.
“Nuisance variables”
Another way to think about random effects is as “nuisance variables”. When modelling something as complex as why pronunciations change, there are always going to be factors missing from our models for a variety of reasons.
- there are things we know matter, but failed to measure them for some reason.
- there are things we know matter, but we don’t know how to measure them.
- there are things that may not be measureable.
- there are things we don’t realize matter.
If we don’t do our best to account for these “unmodellables”, they could very well mess with our fixed effects estimates. While they are not a perfect solution, random effects can go some way towards soaking up some of these unmodelable variables.
On the flip side, if you have something you know has an effect, include it in the model. For example, we know that gender has a big effect on many changes in progress. You should, therefore, include it in the model instead of hoping that the random effects will account for it. That’s because if there is a big between-subject variable that affects the outcome variable, the assumption that speaker random intercepts & slopes are drawn from a single normal distribution is violated.
Pitfalls in mixed effects modelling
The biggest issue people face when fitting a mixed effects model is that it may “fail to converge”. This is because these models aren’t ‘solved’ in the same way as the regular linear regression is. Because of their more complicated nature, these models are usually etimated via a process of iterative optimization.
It is very important to only base your inference on converged models. There are a handful of tweaks that might help in your models converging.
- center and scale all continuous predictors.
- if possible, choose a different factor level that has more data as your reference level.
- remove the correlation component from your model
The imporance of centering and scaling your continuous predictors are very important here. Let’s look at a relatively simple model which is trying to predict F1 by date of birth, with a random intercept by speaker, and a random intercept and slope by word.
ay_fail <- lmer(F1_n ~ dob + (1 | idstring) + (1 + dob | word),
data = ayT)
Model failed to converge with max|grad| = 54.1947 (tol = 0.002, component 1)Model is nearly unidentifiable: very large eigenvalue
- Rescale variables?;Model is nearly unidentifiable: large eigenvalue ratio
- Rescale variables?
summary(ay_fail)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ dob + (1 | idstring) + (1 + dob | word)
Data: ayT
REML criterion at convergence: 28433.3
Scaled residuals:
Min 1Q Median 3Q Max
-5.8279 -0.5969 -0.0404 0.5465 9.6958
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 5.638e-02 0.2374365
word (Intercept) 2.931e-01 0.5413468
dob 1.044e-07 0.0003232 -0.90
Residual 2.410e-01 0.4909313
Number of obs: 19342, groups: idstring, 326; word, 255
Fixed effects:
Estimate Std. Error t value
(Intercept) 23.4652160 1.2169449 19.28
dob -0.0116823 0.0006259 -18.66
Correlation of Fixed Effects:
(Intr)
dob -1.000
convergence code: 0
Model failed to converge with max|grad| = 54.1947 (tol = 0.002, component 1)
Model is nearly unidentifiable: very large eigenvalue
- Rescale variables?
Model is nearly unidentifiable: large eigenvalue ratio
- Rescale variables?
This is bad. But remember, the dob variable is looking at the data like this:
This can pose a challenge for trying to estimate random intercepts and slopes! If we refit the model, this time with the centered and scaled decade predictor, it’ll converge.
ay_win <- lmer(F1_n ~ decade + (1 | idstring) + (1 + decade | word),
data = ayT)
summary(ay_win)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ decade + (1 | idstring) + (1 + decade | word)
Data: ayT
REML criterion at convergence: 28302.8
Scaled residuals:
Min 1Q Median 3Q Max
-5.9964 -0.5958 -0.0401 0.5501 9.5362
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 0.031817 0.17837
word (Intercept) 0.064517 0.25400
decade 0.000555 0.02356 -0.07
Residual 0.240948 0.49086
Number of obs: 19342, groups: idstring, 326; word, 255
Fixed effects:
Estimate Std. Error t value
(Intercept) 1.239348 0.036844 33.64
decade -0.110736 0.006342 -17.46
Correlation of Fixed Effects:
(Intr)
decade -0.714
But sometimes, no matter how you scale or center your predictors, the random effects structure might be too complex given the data. I tried fitting a model like this with the /ayT/ data, but it actually converged:
ay_big <- lmer(F1_n ~ decade * dur_c +
(1 + dur_c | idstring) +
(1 + decade * dur_c | word),
data = ayT)
summary(ay_big)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ decade * dur_c + (1 + dur_c | idstring) + (1 + decade * dur_c | word)
Data: ayT
REML criterion at convergence: 26105.8
Scaled residuals:
Min 1Q Median 3Q Max
-6.4098 -0.5848 -0.0401 0.5324 10.4415
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 0.0279960 0.16732
dur_c 0.0159195 0.12617 0.21
word (Intercept) 0.0920593 0.30341
decade 0.0006019 0.02453 -0.56
dur_c 0.0415790 0.20391 -0.78 0.33
decade:dur_c 0.0017033 0.04127 0.81 -0.38 -0.97
Residual 0.2119319 0.46036
Number of obs: 19342, groups: idstring, 326; word, 255
Fixed effects:
Estimate Std. Error t value
(Intercept) 1.115754 0.040275 27.703
decade -0.102282 0.006440 -15.883
dur_c 0.307936 0.042315 7.277
decade:dur_c 0.001809 0.008129 0.223
Correlation of Fixed Effects:
(Intr) decade dur_c
decade -0.781
dur_c -0.406 0.241
decade:dr_c 0.398 -0.245 -0.914
But often, a model like this will fail to converge, and it’s easy enough to understand why. For many levels of word, there is only 1 token, but the model is trying to model a random intercept, slope for decade and duration, their interaction, and their covariances, for every level of word. One thing that could make a big model like this tractible is to eliminate the covariance structure using the (parameter || group) syntax
ay_bigish <- lmer(F1_n ~ decade * dur_c +
(1 + dur_c | idstring) +
(1 + decade * dur_c || word),
data = ayT)
summary(ay_bigish)
Linear mixed model fit by REML ['lmerMod']
Formula: F1_n ~ decade * dur_c + (1 + dur_c | idstring) + ((1 | word) +
(0 + decade | word) + (0 + dur_c | word) + (0 + decade:dur_c | word))
Data: ayT
REML criterion at convergence: 26118.5
Scaled residuals:
Min 1Q Median 3Q Max
-6.4181 -0.5847 -0.0400 0.5323 10.4281
Random effects:
Groups Name Variance Std.Dev. Corr
idstring (Intercept) 2.805e-02 0.167467
dur_c 1.612e-02 0.126984 0.20
word (Intercept) 5.037e-02 0.224429
word.1 decade 4.893e-04 0.022120
word.2 dur_c 4.365e-03 0.066070
word.3 decade:dur_c 7.132e-05 0.008445
Residual 2.122e-01 0.460683
Number of obs: 19342, groups: idstring, 326; word, 255
Fixed effects:
Estimate Std. Error t value
(Intercept) 1.117453 0.035017 31.91
decade -0.102398 0.006079 -16.85
dur_c 0.288273 0.030780 9.37
decade:dur_c 0.004922 0.005585 0.88
Correlation of Fixed Effects:
(Intr) decade dur_c
decade -0.718
dur_c -0.133 0.088
decade:dr_c 0.068 -0.071 -0.779
We’ll talk more about issues like this more next week when we talk about builing up models.
Type I and Type II Error
These are poorly named, and I think few people remember which is which correctly the first time. These are errors related to the use of p-values. First it’s important to define what the Null Hypothesis is.
- The Null Hypothesis is the Dull Hypothesis: The Null Hypothesis is that there is no relationship between the outcome and predictors. It is not what you think the most likely relationship is.
With that in mind, we can define Type I and Type II errors.
- Type I - You reject the null hypothesis (because p < 0.05), but in reality there isn’t a relationship. That is, you say there is a reliable effect when there isn’t one.
- Type II - You fail to reject the null hypotehsis when there is an real effect.
Or to put it another way, when you see a statistical result that is significant (p < 0.05), but you don’t think it could possibly be real, you think a Type I error has occurred.
And when you get p > 0.05, but you really think there’s something there so you put “trending towards significance” in your paper, you think a Type II error has occurred.
The best way to ameliorate Type I and Type II errors is to either collect more data, or to move to Bayesian Statistics, which we won’t be able to cover in this course, but I recommend this book.
Type M and Type S errors
These are error types discussed by Gelman and colleagues (and are better named).
- Type M Errors are gross over-estimation of the magnitude of an effect size.
- Type S Errors get the sign wrong on an estimate. That is, they predict an effect in the opposite direction of what the true effect is.
Type M might not sound like a very big problem, and Type S might sound a bit outlandish but these can both happen, especially when we are exploring for small effects and filtering results by their p-values. Gelman has proposed that Type M errors are a likely culprit in the so-called “decline effect”, which is a phenomenon some have found that large and significant effects gradually get smaller after their original reports.
Here are some ways to guard youself against Type M and Type S errors.
- Try to reason how big the effect is likely to be.
- Try to collect of how other factors effect the outcome you’re investigating, and how big those effects are.
- Commit to the direction of the effect.
Type M and Type S errors are only going to grow in severity as the size of data sets grow.
Collinearity
“Collinearity” is when two potential predictors are highly correlated with each other. When two correlated predictors are included in the same regression, the results can be wonky at best. Gorman (2010) talks about this at length with respect to socioeconomic measures in the Language Change and Variation study. For example, speakers’ Occupation was highly correlated with the value of their Residence. You shouldn’t include these both, untreated, into your models.
There are at least three different strategies for dealing with this issue
- If you must model these factors independently, residualize (Gorman, 2010)
- Or, just pick the one that is theoretically motivated (Fruehwald, 2016).
- Or give up on treating your measures as having independent reality, realize they are reflections of more abstract dimensions, and try to figure out what those are (i.e. factor analysis, Walker (2016)).
Residualization
Like the name suggests, this involves the residuals from a linear model. If you really want to fit a model like this:
negative_concord ~ occupation + residence
Then what you need to do first is fit this model.
residence ~ occupation
And replace the residence term in the first model with the residuals of the second model. Now, it does matter which order you residualize in. How do you choose? Again, there’s no hard and fast answer, but please don’t try it every possible way and then just choose the model that looks best!
Factor Analysis
Another approach would be to abandon trying to look at all of these predictors independently and to treat them all as rough measures correlated to a more abstract variable, using something like a factor analysis. This is what Walker (2016) did.
The best way to explain a factor analysis is to imagine surveying a bunch of people on the street and asking them the following questions:
- Did you have the heat on this morning?
- Are you wearing a wooly jumper?
- Are you wearing a scarf?
- Could you see your breath when you stepped outside?
- Did your glasses fog up when you walked inside?
- Did you make yourself a big breakfast?
- Was your stomach growling when you woke up?
This looks like you’ve asked people 7 different questions, but really, you’ve only effectively asked 2 questions:
- Was it cold?
- Were you hungry?
If you took the raw data from the 7 question survey and ran it through a factor analysis, it would (ideally) tell you that there are only 2 effective questions in it, based on how responses to questions 1 thru 5 are correlated with each other, and 6&7 are correlated with each other.
Collinearity often comes up as a pitfall of of estimating effects. But it’s also an issue when trying to understand our models. A common, annoying, and usually valid question is:
Couldn’t your effect of A actually be an effect of B?
The best way to deal with this kind of collinearity is to try to kill your effects. Does the same effect hold in the same direction for important subsets of the data? Can you subset the data according to B and fit separate models for A?
LS0tCnRpdGxlOiAiTW9yZSBPbiBNaXhlZCBFZmZlY3RzIE1vZGVscyIKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICBjb2RlX2ZvbGRpbmc6IG5vbmUKICAgIGNzczogLi4vY3VzdG9tLmNzcwogICAgdGhlbWU6IGZsYXRseQogICAgdG9jOiB5ZXMKICAgIHRvY19kZXB0aDogMwogICAgdG9jX2Zsb2F0OiB5ZXMKLS0tCgojIEFnZW5kYQoKLSBSZWNhcHBpbmcgbGFzdCB3ZWVrCi0gRGlzY3Vzc2luZyB0aGUgYXNzdW1wdGlvbnMgb2YgTE1FcwotIFNob3VsZCB2YXJpYWJsZSBiZSBmaXhlZCBvciByYW5kb20gZWZmZWN0cz8KLSBTb21lIHBpdGZhbGxzLgoKCmBgYHtyfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeShicm9vbSkKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KGxtZTQpCmxpYnJhcnkobXVsdGl2YXJpYXRlMjAxNykKYGBgCgpgYGB7cn0KYXlUIDwtIGF5ICU+JSAKICAgICAgICAgIGZpbHRlcihwbHRfdmNsYXNzID09ICJheTAiKSAlPiUKICAgICAgICAgIG11dGF0ZShkb2IgPSB5ZWFyLWFnZSwKICAgICAgICAgICAgICAgICBkZWNhZGUgPSAoZG9iIC0gMTkwMCkvMTAsCiAgICAgICAgICAgICAgICAgbG9nMmR1ciA9IGxvZzIoZHVyKSwKICAgICAgICAgICAgICAgICBkdXJfYyA9IGxvZzJkdXIgLSBtZWRpYW4obG9nMmR1cikpIAoKYGBgCgoKIyBSZWNhcHBpbmcgTE1FIAoKTGV0J3MgcmVjYXAgd2hhdCB3ZSBsZWFybmVkIGFib3V0IGFib3V0IG1peGVkIGVmZmVjdCBtb2RlbHMgbGFzdCB3ZWVrLiAKRmlyc3QsIGl0IGlzIGEgcmVhc29uYWJsZSB0aGluZyB0byB3YW50IHRvIGtub3cgdGhhdCB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gZHVyYXRpb24gYW5kIHRoZSBGMSBvZiAvYXkvIGlzOgoKYGBge3J9CmF5VCAlPiUKICBnZ3Bsb3QoYWVzKGR1cl9jLCBGMV9uKSkrCiAgICBnZW9tX3BvaW50KCkrCiAgICBzdGF0X3Ntb290aChtZXRob2QgPSAnbG0nKSsKICAgIHRoZW1lX2J3KCkKYGBgCgpCdXQgaWYgd2UgZml0IGp1c3QgYSBmbGF0IG1vZGVsLCB0aGVyZSBhcmUgbWFueSBkaWZmZXJlbnQgdmlvbGF0aW9ucyBvZiBzdGF0aXN0aWNhbCBhc3N1bXB0aW9ucywgYW5kIHRoZSBtb2RlbCBpcyBvdmVyY29uZmlkZW50LgoKYGBge3J9CmF5VF9mbGF0X21vZGVsIDwtIGxtKEYxX24gfiBkdXJfYywgZGF0YSA9IGF5VCkKc3VtbWFyeShheVRfZmxhdF9tb2RlbCkKYGBgCgpUaGVyZSBpcyBhIGxvdCBvZiAqc3RydWN0dXJlKiB0byB0aGUgZGF0YSB0aGF0IGlzIGJlaW5nIG92ZXJsb29rZWQgaW4gZXN0aW1hdGluZyB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gZHVyYXRpb24gYW5kIEYxLiAKRmlyc3Qgb2YgYWxsLCBpdCBkb2Vzbid0IGxvb2sgbGlrZSBldmVyeSBzcGVha2VyIGhhcyB0aGUgc2FtZSByZWxhdGlvbnNoaXAuCgpgYGB7cn0KcmFuZG9tX3NwZWFrZXJzIDwtIHNhbXBsZSh1bmlxdWUoYXlUJGlkc3RyaW5nKSwgOCkKYXlUJT4lCiAgZmlsdGVyKGlkc3RyaW5nICVpbiUgcmFuZG9tX3NwZWFrZXJzKSU+JQogIGdncGxvdChhZXMoZHVyX2MsIEYxX24pKSsKICAgIGdlb21fcG9pbnQoKSsKICAgIHN0YXRfc21vb3RoKG1ldGhvZCA9IGxtKSsKICAgIGZhY2V0X3dyYXAofmlkc3RyaW5nLCBuY29sPSA0KSsKICAgIHRoZW1lX2J3KCkKYGBgCgpTZWNvbmQsIHRoZSBkYXRhIGlzICpmYXIqIGZyb20gYmFsYW5jZWQgYWNyb3NzIHNwZWFrZXJzLgoKYGBge3J9CmF5VCAlPiUKICBjb3VudChpZHN0cmluZyklPiUKICBnZ3Bsb3QoYWVzKG4pKSsKICAgIHN0YXRfYmluKCkrCiAgICBzY2FsZV94X2xvZzEwKCkrCiAgICB4bGFiKCJuIChsb2cgc2NhbGUpIikrCiAgICB0aGVtZV9idygpCmBgYAoKCkxldCdzIGxvb2sgYWdhaW4gYXQgb3VyIGVzdGltYXRlcyBmcm9tIHRoZSBmbGF0IG1vZGVsOgoKYGBge3J9CnN1bW1hcnkoYXlUX2ZsYXRfbW9kZWwpJGNvZWYKYGBgCgpUaG9zZSBlc3RpbWF0ZXMgYXJlIGdvaW5nIHRvIGxhcmdlbHkgYmUgYSBjaGFyYWN0ZXJpemF0aW9uIG9mIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiBkdXJhdGlvbiBhbmQgRjEgKmZvciB0aGUgc3BlYWtlcnMgd2l0aCB0aGUgbW9zdCBkYXRhKi4KT25lIHdheSwgeW91IGNvdWxkIHRyeSB0byBnZXQgYXJvdW5kIHRoaXMgcHJvYmxlbSB3b3VsZCBiZSB0byBmaXQgb25lIGxpbmVhciBtb2RlbCBmb3IgZXZlcnkgc3BlYWtlci4KCmBgYHtyfQpheVQlPiUKICBnZ3Bsb3QoYWVzKGR1cl9jLCBGMV9uKSkrCiAgICBzdGF0X3Ntb290aChtZXRob2QgPSBsbSwgCiAgICAgICAgICAgICAgICBhZXMoZ3JvdXAgPSBpZHN0cmluZyksIAogICAgICAgICAgICAgICAgc2UgPSBGLAogICAgICAgICAgICAgICAgZ2VvbSA9ICJsaW5lIiwKICAgICAgICAgICAgICAgIGFscGhhID0gMC40KSsKICAgIHRoZW1lX2J3KCkKYGBgCgpCdXQgdGhpcyBpc24ndCBpZGVhbCBmb3IgYSBmZXcgcmVhc29ucy4gCgotIFdlIGRvbid0IHdpbmQgdXAgd2l0aCBhIHByZWNpc2UgZXN0aW1hdGUgb2YgdGhlIGdlbmVyYWwgZWZmZWN0IG9mIGR1cmF0aW9uIG9uIEYxLgotIFdlIGRvbid0IHdpbmQgdXAgd2l0aCBhbiBlc3RpbWF0ZSBvZiBvdXIgdW5jZXJ0YWludHkgb2YgdGhlIGVmZmVjdCBvZiBkdXJhdGlvbiBvbiBGMS4KLSBOb25lIG9mIHRoZSBlc3RpbWF0ZXMgb2YgdGhlc2UgbW9kZWxzIHRha2VzIGludG8gYWNjb3VudCB0aGUgZXN0aW1hdGVzIG9mIGFueSBvdGhlciBtb2RlbC4KClNvLCBpbnN0ZWFkLCB3ZSBmaXQgYSBtaXhlZCBlZmZlY3RzIG1vZGVsIHdpdGggYSByYW5kb20gaW50ZXJjZXB0IGFuZCBhIHJhbmRvbSBzbG9wZSBieSBzcGVha2VyLgoKYGBge3J9CmF5VF9sbWUgPC0gbG1lcihGMV9uIH4gZHVyX2MgKyAoMSArIGR1cl9jIHwgaWRzdHJpbmcpLCBkYXRhID0gYXlUKQpzdW1tYXJ5KGF5VF9sbWUpCmBgYAoKSGVyZSwgd2UgaGF2ZSB0aGUgZml4ZWQgZWZmZWN0cyBlc3RpbWF0ZXM6CgpgYGB7cn0Kc3VtbWFyeShheVRfbG1lKSRjb2VmCmBgYAoKVGhpcyBpcyB0aGUgYmVzdCAqZXN0aW1hdGUgYWNyb3NzIHNwZWFrZXJzKiBmb3IgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIEYxIGFuZCBkdXJhdGlvbi4KQnV0IHdlIGFsc28gZ2V0IHNvbWUgdmFsdWFibGUgaW5mbyBhYm91dCBob3cgdmFyaWFibGUgKmFyb3VuZCogdGhlc2UgZXN0aW1hdGVzIHdlIGNhbiBleHBlY3Qgb2JzZXJ2ZWQgc3BlYWtlcnMgdG8gYmUhCgpgYGB7cn0Kc3VtbWFyeShheVRfbG1lKSR2YXJjb3IKYGBgCgpUaGUgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIHRoZSByYW5kb20gc2xvcGVzIGhlcmUgaXMgZmFpcmx5IHJlc3BlY3RhYmxlLgpMZXQncyBzYXkgdGhlcmUgd2FzIGEgc3BlYWtlciB3aG8gaGFkICpkb3VibGUqIHRoZSBzbG9wZSBvZiB0aGUgZ3JvdXAgZXN0aW1hdGUuCkhvdyBtYW55IHN0YW5kYXJkIGRldmlhdGlvbnMgd291bGQgdGhleSBiZSBmcm9tIHRoZSBncm91cCBhdmVyYWdlPwoKCmBgYHtyfQpzbG9wZSA8LSAwLjM0MjgzOTIgCnNsb3BlX3NkIDwtIDAuMTMzMDEKCigoc2xvcGUgKiAyKSAtIHNsb3BlKS9zbG9wZV9zZApgYGAKClR3byBhbmQgYSBoYWxmIHNkcyBpcyBmYWlybHkgbGFyZ2UsIGJ1dCBub3QgYW4gZXh0cmVtZSBvdXRsaWVyLgoKCgojIFRoZSBhc3N1bXB0aW9ucyBvZiBNaXhlZCBFZmZlY3RzIE1vZGVscwoKTWl4ZWQgZWZmZWN0cyBtb2RlbHMgcmVsYXggc29tZSBhc3N1bXB0aW9ucyBvZiBmaXhlZC1lZmZlY3RzIG1vZGVscywgbmFtZWx5IHRoYXQgYWxsIG9ic2VydmF0aW9ucyBhcmUgaS5pLmQuIEl0IGRvZXMgc28gYnkgYWxsb3dpbmcgZm9yIHBhcnRpYWwgcG9vbGluZyBvZiB0aGUgZGF0YSBhY2NvcmRpbmcgdG8gZ3JvdXBpbmcgZmFjdG9ycy4KSG93ZXZlciwgbWl4ZWQgZWZmZWN0cyBtb2RlbHMgYXJlIG5vdCAqd2l0aG91dCogYXNzdW1wdGlvbnMuCgpTcGVjaWZpY2FsbHksIHRoZSBtb2RlbCBhc3N1bWVzIHRoYXQgKnRoZSByYW5kb20gZWZmZWN0cyBhcmUgZHJhd24gZnJvbSBhIG5vcm1hbCBkaXN0cmlidXRpb24qLgpUaGUgY2x1ZSB0aGF0IHRoZSBtb2RlbCBpcyBtYWtpbmcgdGhpcyBhc3N1bXB0aW9uIGlzIGluIHRoYXQgaXQgZ2l2ZXMgdXMgZXN0aW1hdGVzIG9mIHN0YW5kYXJkIGRldmlhdGlvbiBvZiB0aGUgcmFuZG9tIGVmZmVjdHMsIHdoaWNoIGlzIGEgcHJvcGVydHkgb2Ygbm9ybWFsIGRpc3JpYnV0dW9ucy4KClRoaXMgYXNzdW1wdGlvbiBoYXMgYSBnb29kIGVmZmVjdCB3aGVuIGl0IGNvbWVzIHRvIHByZWRpY3RpbmcgdGhlIHNsb3BlcyBhbmQgaW50ZXJjZXB0cyBmb3IgaW5kaXZpZHVhbCBzcGVha2Vycy4KTGFzdCB3ZWVrLCB3ZSBsb29rZWQgYXQgdGhlIHJlc3VsdHMgb2YgZml0dGluZyBvbmUgbGluZWFyIG1vZGVsIGZvciBlYWNoIHNwZWFrZXIuCkhlcmUncyB0aGUgZGVuc2l0eSBwbG90cyBvZiB0aGUgZGlzdHJpYnV0aW9uIG9mIGFsbCBvZiB0aGUgc2xvcGVzIGFuZCBpbnRlcmNlcHRzIGZyb20gYWxsIG9mIHRob3NlIG1vZGVscy4KCmBgYHtyIGVjaG8gPSBGfQpheVQgJT4lIAogIGdyb3VwX2J5KGlkc3RyaW5nKSU+JQogIG5lc3QoKSU+JQogIG11dGF0ZShtb2RlbCA9IG1hcChkYXRhLCB+bG0oRjFfbiB+IGR1cl9jLCBkYXRhID0gLngpKSwKICAgICAgICAgcGFyYW1zID0gbWFwKG1vZGVsLCB0aWR5KSwKICAgICAgICAgbiA9IG1hcChkYXRhLCBucm93KSAlPiUgc2ltcGxpZnkoKSklPiUKICB1bm5lc3QocGFyYW1zKSU+JQogIHNlbGVjdChpZHN0cmluZywgdGVybSwgZXN0aW1hdGUsIG4pJT4lCiAgc3ByZWFkKHRlcm0sIGVzdGltYXRlKSAtPiBwYXJhbXMKCmF5VF9sbWVfZml4ZWYgPC0gZml4ZWYoYXlUX2xtZSkKCmBgYAoKYGBge3IgZWNobyA9IEZ9CnBhcmFtcyAlPiUKICBnYXRoZXIocGFyYW0sIGVzdCwgYChJbnRlcmNlcHQpYDpkdXJfYyktPnBhcmFtX2xvbmcKcGFyYW1fbG9uZyAlPiUgCiAgZ2dwbG90KGFlcyhlc3QpKSsKICAgIHN0YXRfZGVuc2l0eShhbHBoYSA9IDAuOCwgY29sb3IgPSAiYmxhY2siKSsKICAgIGZhY2V0X3dyYXAofnBhcmFtLCBzY2FsZXMgPSAiZnJlZSIpKwogICAgeGxhYigiZXN0aW1hdGUiKSsKICAgIHRoZW1lX2J3KCkKYGBgCgoKCk5vdywgbGV0J3MgY29tcGFyZSB0aGVzZSBieS1zcGVha2VyIGludGVyY2VwdHMgYW5kIHNsb3BlcyAoaW4gZ3JleSkgdG8gdGhlIHJhbmRvbSBlZmZlY3RzIGZyb20gb3VyIGxpbmVhciBtaXhlZCBlZmZlY3RzIG1vZGVsIChpbiB5ZWxsb3cpLgpgYGB7ciBlY2hvID0gRn0KbGlicmFyeShnZ3RoZW1lcykKcmFuZWYoYXlUX2xtZSkkaWRzdHJpbmcgJT4lCiAgbXV0YXRlKGAoSW50ZXJjZXB0KWAgPSBgKEludGVyY2VwdClgKyBheVRfbG1lX2ZpeGVmWzFdLAogICAgICAgICAgZHVyX2MgPSBkdXJfYyArIGF5VF9sbWVfZml4ZWZbMl0sCiAgICAgICAgIGVzdGltYXRvciA9ICJsbWUiKSU+JQogIGdhdGhlcihwYXJhbSwgZXN0LCBgKEludGVyY2VwdClgOmR1cl9jKSU+JQogIGJpbmRfcm93cyhwYXJhbV9sb25nICU+JSBtdXRhdGUoZXN0aW1hdG9yID0gImxtIikpJT4lCiAgZ2dwbG90KGFlcyhlc3QpKSsKICAgIGdlb21fZGVuc2l0eShhZXMoZmlsbCA9IGVzdGltYXRvciksIAogICAgICAgICAgICAgICAgIHBvc2l0aW9uID0gImlkZW50aXR5IiwKICAgICAgICAgICAgICAgICBhbHBoYSA9IDAuNikrCiAgICBmYWNldF93cmFwKH5wYXJhbSwgc2NhbGVzPSJmcmVlIikrCiAgICBzY2FsZV9maWxsX2NvbG9yYmxpbmQoKSsKICAgIHRoZW1lX2J3KCkKYGBgCgoKWW91IGNhbiBzZWUgdGhhdCBmb3IgYm90aCB0aGUgaW50ZXJjZXB0IGFuZCB0aGUgc2xvcGUsIGJ1dCBlc3BlY2lhbGx5IHRoZSBzbG9wZSwgdGhlIHJhbmdlIG9mIGludGVyc3BlYWtlciB2YWx1ZXMgaGFzIGJlZW4gc3Vja2VkIGluLgpUaGlzIGlzIGFsc28ga25vd24gYXMgInNocmlua2FnZS4iCklmIHdlIHdlcmUgaW50ZXJlc3RlZCBpbiBob3cgZGlmZmVyZW50IHRoZSBlZmZlY3Qgb2YgZHVyYXRpb24gY291bGQgYmUgYmV0d2VlbiBzcGVha2VycywgdGhpcyB3b3VsZCBzdWdnZXN0IHRoYXQgc2ltcGx5IGZpdHRpbmcgYSBtb2RlbCBmb3IgZXZlcnkgc3BlYWtlciB3b3VsZCAqb3ZlcmVzdGltYXRlKiB0aGUgdmFyaWFuY2UhCgpTaHJpbmthZ2UgaXMgZGVzaXJlYWJsZSBmb3IgYSBmZXcgcmVhc29ucy4KVGhlIGZpcnN0IGlzIHRoYXQgaXQgaXMgb2Z0ZW4gdGhlIGNhc2UgdGhhdCBleHRyZW1lIG91dGxpZXIgc3ViamVjdHMgd2lsbCBiZSBsZXNzIGV4dHJlbWUgb24gYSBzZWNvbmQgdGVzdGluZyAoaS5lLiByZWdyZXNzaW9uIHRvIHRoZSBtZWFuKS4gClNvLCBpdCBtYWtlcyBzZW5zZSB0byBtb2RlcmF0ZSB0aGUgZXN0aW1hdGVzIG9mIG91dGxpZXIgc3BlYWtlcnMuClRoZSBzZWNvbmQgaXMgdGhhdCB3ZSBvdWdodCB0byBoYXZlIGhpZ2hlciB1bmNlcnRlcnRhaW50eSBmb3IgbWFueSBvZiB0aGUgc3BlYWtlcnMgd2hvIGFyZSBlc3RpbWF0ZWQgdG8gaGF2ZSBhbiBleHRyZW1lIGR1cmF0aW9uIGVmZmVjdCB0aGFuIGZvciB0aG9zZSBzcGVha2VycyB3aG9zZSBzbG9wZXMgYXJlIGNsb3NlciB0byB0aGUgZ3JvdXAgYXZlcmFnZQoKClRvIGRlbW9uc3RyYXRlIHRoaXMgcG9pbnQsIGhlcmUgYXJlIHRoZSBzaXggc3BlYWtlcnMgd2hvIGhhZCB0aGUgYmlnZ2VzdCBkaWZmZXJlbmNlIGJldHdlZW4gdGhlICpNYXhpbXVtIExpa2VsaWhvb2QqIGVzdGltYXRlIG9mIHRoZWlyIHNsb3BlcyAodGhhdCBpcywgZnJvbSBmaXR0aW5nIGEgbW9kZWwgZm9yIGVhY2ggc3BlYWtlcikgYW5kIHRoZWlyIHByZWRpY3RlZCBzbG9wZXMgZnJvbSB0aGUgcmFuZG9tIGVmZmVjdHMuClRoZSBNYXhpbXVtIExpa2VsaWhvb2QgZXN0aW1hdGUgaXMgdGhlIGRhc2hlZCBibGFjayBsaW5lLCBhbmQgdGhlIHJhbmRvbSBlZmZlY3RzIHByZWRpY3Rpb25zIGFyZSBpbiB0aGUgcmVkIGxpbmVzLgoKCmBgYHtyIGVjaG8gPSBGfQpheVRfbG1lX3JhbmVmIDwtIHJhbmVmKGF5VF9sbWUpJGlkc3RyaW5nCmF5VF9sbWVfcmFuZWYkaWRzdHJpbmcgPC0gcm93bmFtZXMoYXlUX2xtZV9yYW5lZikKYXlUX2xtZV9yYW5lZiA8LSBheVRfbG1lX3JhbmVmICU+JSAKICAgICAgICAgICAgICAgICAgICBtdXRhdGUoYChJbnRlcmNlcHQpYCA9IGAoSW50ZXJjZXB0KWAgKyBheVRfbG1lX2ZpeGVmWzFdLAogICAgICAgICAgICAgICAgICAgICAgICAgICBkdXJfYyA9IGR1cl9jICsgYXlUX2xtZV9maXhlZlsyXSkKYXlUX2xtZV9yYW5lZl9sb25nIDwtIGF5VF9sbWVfcmFuZWYgJT4lIAogICAgICAgICAgICAgICAgICAgICAgICAgIGdhdGhlcihwYXJhbSwgZXN0LCAxOjIpJT4lCiAgICAgICAgICAgICAgICAgICAgICAgICAgbXV0YXRlKG1vZGVsID0gImxtZSIpCnBhcmFtX2xvbmcgJT4lIAogIG11dGF0ZShtb2RlbCA9ICJsbSIpJT4lCiAgYmluZF9yb3dzKGF5VF9sbWVfcmFuZWZfbG9uZyklPiUKICBzZWxlY3QoaWRzdHJpbmcsIHBhcmFtLCBlc3QsIG1vZGVsKSU+JQogIHNwcmVhZChtb2RlbCwgZXN0KSU+JQogIGZpbHRlcihwYXJhbSA9PSAnZHVyX2MnKSU+JQogIG11dGF0ZShkaWZmID0gYWJzKGxtLWxtZSkpJT4lCiAgYXJyYW5nZSgtZGlmZikgJT4lCiAgc2xpY2UoMTo2KS0+ZXh0cmVtYV9kaWZmCgoKcGFyYW1fbG9uZyAlPiUgCiAgbXV0YXRlKG1vZGVsID0gImxtIiklPiUKICBiaW5kX3Jvd3MoYXlUX2xtZV9yYW5lZl9sb25nKSU+JQogIHNlbGVjdChpZHN0cmluZywgcGFyYW0sIGVzdCwgbW9kZWwpJT4lCiAgc3ByZWFkKG1vZGVsLCBlc3QpJT4lCiAgZmlsdGVyKHBhcmFtID09ICdkdXJfYycpJT4lCiAgbXV0YXRlKGRpZmYgPSBhYnMobG0tbG1lKSklPiUKICBhcnJhbmdlKGRpZmYpICU+JQogIHNsaWNlKDE6NiktPmV4dHJlbWFfc2FtZQoKYXlUICU+JQogIGZpbHRlcihpZHN0cmluZyAlaW4lIGV4dHJlbWFfZGlmZiRpZHN0cmluZyklPiUKICBtdXRhdGUoaWRzdHJpbmcgPSBmYWN0b3IoaWRzdHJpbmcsIGxldmVscyA9IGV4dHJlbWFfZGlmZiRpZHN0cmluZykpJT4lCiAgZ2dwbG90KGFlcyhkdXJfYywgRjFfbikpKwogICAgZ2VvbV9wb2ludCgpKwogICAgZ2VvbV9hYmxpbmUoZGF0YSA9IHBhcmFtcyAlPiUgZmlsdGVyKGlkc3RyaW5nICVpbiUgZXh0cmVtYV9kaWZmJGlkc3RyaW5nKSwKICAgICAgICAgICAgICAgIGFlcyhzbG9wZSA9IGR1cl9jLAogICAgICAgICAgICAgICAgICAgIGludGVyY2VwdCA9IGAoSW50ZXJjZXB0KWApLAogICAgICAgICAgICAgICAgbGluZXR5cGUgPSAyKSsKICAgIGdlb21fYWJsaW5lKGRhdGEgPSBheVRfbG1lX3JhbmVmICU+JSBmaWx0ZXIoaWRzdHJpbmcgJWluJSBleHRyZW1hX2RpZmYkaWRzdHJpbmcpLAogICAgICAgICAgICAgICAgYWVzKHNsb3BlID0gZHVyX2MsCiAgICAgICAgICAgICAgICAgICAgaW50ZXJjZXB0ID0gYChJbnRlcmNlcHQpYCksCiAgICAgICAgICAgICAgICBjb2xvciA9ICJyZWQiKSsgIAogICAgZmFjZXRfd3JhcCh+aWRzdHJpbmcpKwogICAgdGhlbWVfYncoKQoKYGBgCgpDb21wYXJlIHRoaXMgdG8gdGhlIHNwZWFrZXJzIHdob3NlIE1MIGVzdGltYXRlcyBhcmUgdGhlIG1vc3Qgc2ltaWxhciB0byB0aGUgcmFuZG9tIGVmZmVjdHMgcHJlZGljdGlvbnMuCgpgYGB7ciBlY2hvID0gRn0KYXlUICU+JQogIGZpbHRlcihpZHN0cmluZyAlaW4lIGV4dHJlbWFfc2FtZSRpZHN0cmluZyklPiUKICBtdXRhdGUoaWRzdHJpbmcgPSBmYWN0b3IoaWRzdHJpbmcsIGxldmVscyA9IGV4dHJlbWFfc2FtZSRpZHN0cmluZykpJT4lCiAgZ2dwbG90KGFlcyhkdXJfYywgRjFfbikpKwogICAgZ2VvbV9wb2ludCgpKwogICAgZ2VvbV9hYmxpbmUoZGF0YSA9IHBhcmFtcyAlPiUgZmlsdGVyKGlkc3RyaW5nICVpbiUgZXh0cmVtYV9zYW1lJGlkc3RyaW5nKSwKICAgICAgICAgICAgICAgIGFlcyhzbG9wZSA9IGR1cl9jLAogICAgICAgICAgICAgICAgICAgIGludGVyY2VwdCA9IGAoSW50ZXJjZXB0KWApLAogICAgICAgICAgICAgICAgbGluZXR5cGUgPSAyKSsKICAgIGdlb21fYWJsaW5lKGRhdGEgPSBheVRfbG1lX3JhbmVmICU+JSBmaWx0ZXIoaWRzdHJpbmcgJWluJSBleHRyZW1hX3NhbWUkaWRzdHJpbmcpLAogICAgICAgICAgICAgICAgYWVzKHNsb3BlID0gZHVyX2MsCiAgICAgICAgICAgICAgICAgICAgaW50ZXJjZXB0ID0gYChJbnRlcmNlcHQpYCksCiAgICAgICAgICAgICAgICBjb2xvciA9ICJyZWQiKSsgIAogICAgZmFjZXRfd3JhcCh+aWRzdHJpbmcpKwogICAgdGhlbWVfYncoKQpgYGAKCgpUaGUgYXNzdW1wdGlvbiB0aGF0IHRoZSByYW5kb20gZWZmZWN0cyBhcmUgZHJhd24gZnJvbSBhIG5vcm1hbCBkaXN0cmlidXRpb24gYXJlIGdyZWF0IGJlY2F1c2UgCgoxLiBTaHJpbmthZ2UgaXMgZ29vZC4KMi4gSXQgbWFrZXMgdGhlaXIgZXN0aW1hdGlvbiB0cmFjdGlibGUgb24gbW9zdCBsYXB0b3BzIQoKQnV0IGFzIHdpdGggYWxsIHN0YXRpc3RpY2FsICphc3N1bXB0aW9ucyosIHNvbWUgY2F1dGlvbiBzaG91bGQgYmUgdGFrZW4gd2hlcmUgdGhleSBtYXkgYmUgdmlvbGF0ZWQuClRoZSBtb3N0IG1ham9yLCBhbmQgbGlrZWx5IG1vc3QgY29tbW9uLCB2aW9sYXRpb24gb2YgdGhpcyBub3JtYWxpdHkgYXNzdW1wdGlvbiBpcyBpZiBhbGwgb2YgdGhlIGxldmVscyBvZiB5b3VyIGdyb3VwaW5nIGZhY3RvciAqYXJlbid0KiBkcmF3biBmcm9tIGEgc2luZ2xlIGRpc3RyaWJ1dGlvbiEKRm9yIGV4YW1wbGUsIGhlcmUgaXMgc29tZSBzaW11bGF0ZWQgZGF0YSBmcm9tIDYgc3BlYWtlcnMuIFNwZWFrZXJzIGBhYCwgYGJgIGFuZCBgY2AgY29tZSBmcm9tIGEgZ3JvdXAgZGlzcmlidXRpb24gd2l0aCBtZWFuIDAsIGFuZCBzcGVha2VycyBgZGAsIGBlYCBhbmQgYGZgIGNvbWUgZnJvbSBhIGdyb3VwIGRpc3RyaWJ1dGlvbiB3aXRoIG1lYW4gNC4KCgpgYGB7ciB0aWR5ID0gRkFMU0V9CnNwZWFrZXJzIDwtIGxldHRlcnNbMTo2XQpzcGVha2VyX21lYW5zIDwtIGMocm5vcm0oMywgbWVhbiA9IDAsIHNkID0gMSksCiAgICAgICAgICAgICAgICAgICBybm9ybSgzLCBtZWFuID0gNCwgc2QgPSAxKSkKYGBgCmBgYHtyfQpzcGVha2VyX3NhbXBzIDwtIG1hcChzcGVha2VyX21lYW5zLCB+cm5vcm0oNTAsIG1lYW4gPSAueCwgc2QgPSAxKSkgJT4lIHNpbXBsaWZ5KCkKYGBgCmBgYHtyfQpmYWtlX3NwZWFrZXJzIDwtIGRhdGEuZnJhbWUoc3BlYWtlciA9IHJlcChzcGVha2VycywgZWFjaCA9IDUwKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdyb3VwID0gcmVwKGMoIngiLCAieSIpLCBlYWNoID0gKDUwICogMykpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZSA9IHNwZWFrZXJfc2FtcHMpCmBgYApgYGB7cn0KZ2dwbG90KGZha2Vfc3BlYWtlcnMsIGFlcyhvdXRjb21lLCBmaWxsID0gZ3JvdXApKSsKICBzdGF0X2Jpbihwb3NpdGlvbiA9ICJpZGVudGl0eSIsIGFscGhhID0gMC44KSsKICBzY2FsZV9maWxsX2NvbG9yYmxpbmQoKSsKICB0aGVtZV9idygpCmBgYAoKV2UgKmNvdWxkKiBmaXQgYW4gbWl4ZWQgZWZmZWN0cyBtb2RlbCB3aXRoIGp1c3QgYSByYW5kb20gaW50ZXJjZXB0IGJ5IHNwZWFrZXIuCgpgYGB7cn0KZmFrZV9sbWVfMSA8LSBsbWVyKG91dGNvbWUgfiAoMXxzcGVha2VyKSwgZGF0YSA9IGZha2Vfc3BlYWtlcnMpCnN1bW1hcnkoZmFrZV9sbWVfMSkKYGBgCgo8ZGl2IGNsYXNzID0gImJveCBicmVhayI+CjxzcGFuIGNsYXNzID0gImJpZy1sYWJlbCI+UXVlc3Rpb248L3NwYW4+CgpXaGF0J3MgdGhlIHByb2JsZW0/CjwvZGl2PgoKVGhlIExNRSBpcyBlc3RpbWF0aW5nIGp1c3QgMSB2YXJpYW5jZSBwYXJhbWV0ZXIgZm9yIGFsbCBvZiB0aGUgc3BlYWtlcnMsIHRodXMgYXNzdW1pbmcgYWxsIHNwZWFrZXJzIGFyZSBkcmF3biBmcm9tIDEgZGlzdHJpYnV0aW9uLCBidXQgdGhlcmUgYXJlIGFjdHVhbGx5ICp0d28qIGRpc3RyaWJ1dGlvbnMgb2Ygc3BlYWtlcnMgaGVyZSEKClRoZXJlIGlzIGEgZmFpcmx5IHN0cmFpZ2h0Zm9yd2FyZHcgd2F5IHRvIGFkZHJlc3MgdGhpcyB2aW9sYXRpb24gb2YgdGhlIExNRSBhc3N1bXB0aW9uLCBhbmQgdGhhdCdzIHRvIGluY2x1ZGUgdGhlIHJlbGV2YW50IGdyb3VwaW5nIHByZWRpY3RvciBpbiB0aGUgbW9kZWwhCgpgYGB7cn0KZmFrZV9sbWVfMiA8LSBsbWVyKG91dGNvbWUgfiBncm91cCArICgxfHNwZWFrZXIpLCAKICAgICAgICAgICAgICAgICAgIGRhdGEgPSBmYWtlX3NwZWFrZXJzKQpgYGAKYGBge3J9CnN1bW1hcnkoZmFrZV9sbWVfMikKYGBgCgpOb3cgdGhhdCB3ZSBoYXZlIHRoZSByZWxldmFudCBiZXR3ZWVuLXNwZWFrZXIgcHJlZGljdG9yIGluY2x1ZGVkIGluIHRoZSBtb2RlbCwgbm90aWNlIGhvdyBtdWNoIHRoZSB2YXJpYW5jZSBvZiB0aGUgcmFuZG9tLWludGVyY2VwdHMgaGFzIGRlY3JlYXNlZC4KClRoaXMgYnJpbmdzIHVwIGFuIGltcG9ydGFudCBwb2ludCAtIElmIHRoZXJlIGlzIGEgY3J1Y2lhbCBiZXR3ZWVuLXNwZWFrZXJzLCBiZXR3ZWVuLXdvcmRzLCBiZXR3ZWVuLXN1YmplY3QsIGJldHdlZW4taXRlbXMsIGJldHdlZW4td2hhdGV2ZXIgcHJlZGljdG9yLCAqaW5jbHVkZSBpdCBpbiB0aGUgbW9kZWwqLgoKUmV0dXJuaW5nIHRvIHRoZSByZWFsIGRhdGEgZXhhbXBsZSBvZiBgYXlUYCwgdGhlcmUgaXMgb25lIG9uZSBtYWpvciBpbnRlcnNwZWFrZXIgdmFyaWFibGUgd2hpY2ggd2FzIG5vdCBpbmNsdWRlZCBpbiB0aGUgbW9kZWwgLS0gZGF0ZSBvZiBiaXJ0aC4KUGxvdHRpbmcgRjEgYWdhaW5zdCBib3RoIGRlY2FkZSBhbmQgZHVyYXRpb24sIHRoZXJlIHNlZW0gdG8gYmUgcmVsaWFibGUgZWZmZWN0cy4KQnV0IGJlY2F1c2UgZGF0ZSBvZiBiaXJ0aCBpcyBhIGJldHdlZW4tc3BlYWtlcnMgcHJlZGljdG9yLCB3aGVuIGl0IHdhc24ndCBpbmNsdWRlZCBpbiB0aGUgbW9kZWwsIHRoaXMgYmV0d2VlbiBzcGVha2VyIHZhcmlhdGlvbiB3YXMgYWNjb3VudGVkIGZvciBieSBieS1zcGVha2VyIHJhbmRvbSBpbnRlcmNlcHRzLiAKCjxkaXYgc3R5bGUgPSAid2lkdGg6MTAwJTtmbG9hdDpsZWZ0OyI+CjxkaXYgc3R5bGUgPSAid2lkdGg6NDUlO2Zsb2F0OmxlZnQ7bWFyZ2luLWxlZnQ6YXV0bzttYXJnaW4tcmlnaHQ6YXV0bzsiPgpgYGB7ciBlY2hvID0gRn0KZ2dwbG90KGF5VCwgYWVzKGRlY2FkZSwgRjFfbikpKwogICAgZ2VvbV9wb2ludCgpKwogICAgdGhlbWVfYncoKSsKICAgIGdndGl0bGUoImRlY2FkZSIpCmBgYAo8L2Rpdj4KPGRpdiBzdHlsZSA9ICJ3aWR0aDo0NSU7IGZsb2F0OmxlZnQ7bWFyZ2luLWxlZnQ6YXV0bzttYXJnaW4tcmlnaHQ6YXV0bzsiPgpgYGB7ciBlY2hvID0gRn0KZ2dwbG90KGF5VCwgYWVzKGR1cl9jLCBGMV9uKSkrCiAgICBnZW9tX3BvaW50KCkrCiAgICB0aGVtZV9idygpKwogICAgZ2d0aXRsZSgiZHVyYXRpb24iKQpgYGAKPC9kaXY+CjwvZGl2PgoKYGBge3J9CmF5VF9sbWVfbXVsdGkgPC0gbG1lcihGMV9uIH4gZGVjYWRlICogZHVyX2MgKyAoMSArIGR1cl9jIHwgaWRzdHJpbmcpLAogICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGF5VCkKc3VtbWFyeShheVRfbG1lX211bHRpKQpgYGAKCgoKIyMgVmlvbGF0aW9ucyBvZiBhc3N1bXB0aW9ucyB0aGF0ICpjYW4ndCogYmUgYWNjb21vZGF0ZWQKClRoZXJlIGFyZSBhIGZldyBjaXJjdW1zdGFuY2VzIHRoYXQgbWlnaHQgKGRlZmluaXRlbHkpIGFyaXNlIGluIHlvdXIgYWN0dWFsIGRhdGEgdGhhdCB2aW9sYXRlIHRoZSBhc3N1bXB0aW9ucyBvZiBtaXhlZCBlZmZlY3RzIG1vZGVscyB0aGF0IHdlIGNhbid0IGRvIGFueXRoaW5nIGFib3V0IHdpdGhpbiB0aGUgZnJhbWV3b3JrLgoKIyMjIERpZmZlcmVudCBncm91cCB2YXJpYW5jZXMKCkZpcnN0LCBsZXQncyByZXZpc2l0ICp3aHkqIGFkZGluZyB0aGUgYmV0d2Vlbi1zcGVha2VyIGdyb3VwaW5nIHByZWRpY3RvciB3YXMgZWZmZWN0aXZlIGluIHNhdGlzZnlpbmcgdGhlIG5vcm1hbGl0eSBhc3N1bXB0aW9uLgoKYGBge3J9CmZha2VfbG1lXzIKYGBgCgpUaGUgd2F5IHRvIHRoaW5rIGFib3V0IHRoZSByYW5kb20gaW50ZXJjZXB0cyBpcyB0aGF0IGFmdGVyIHRha2luZyBpbnRvIGFjY291bnQgYWxsIG9mIHRoZSBmaXhlZCBlZmZlY3RzLCB0aGV5J3JlIGhvdyBtdWNoIHJlbWFpbnMgdG8gYmUgZXhwbGFpbmVkIGJ5IHRoZSBwb29saW5nIHZhcmlhYmxlcy4KV2UgY2FuIHZpc3VhbGl6ZSBob3cgdGhlIHJhbmRvbSBlZmZlY3RzIHN0cnVjdHVyZSAic2VlcyIgdGhlIGRhdGEgYnkgc2ltcGx5IHN1YnRyYWN0aW5nIG91dCB0aGUgZml4ZWQgZWZmZWN0IG9mIGdyb3VwIGB5YCBmcm9tIHRoZWlyIG91dGNvbWUgZGF0YS4KCmBgYHtyfQpmYWtlX3NwZWFrZXJzJG91dGNvbWUyIDwtIGZha2Vfc3BlYWtlcnMkb3V0Y29tZQpmYWtlX3NwZWFrZXJzJG91dGNvbWUyW2Zha2Vfc3BlYWtlcnMkZ3JvdXAgPT0gInkiXSA8LSBmYWtlX3NwZWFrZXJzJG91dGNvbWVbZmFrZV9zcGVha2VycyRncm91cCA9PSAieSJdIC0gZml4ZWYoZmFrZV9sbWVfMilbMl0KYGBgCgpgYGB7cn0KZ2dwbG90KGZha2Vfc3BlYWtlcnMsIGFlcyhvdXRjb21lMikpKwogIHN0YXRfYmluKGFlcyhmaWxsID0gZ3JvdXApLCBwb3NpdGlvbiA9ICJpZGVudGl0eSIsIGFscGhhPSAwLjYpKwogIHNjYWxlX2ZpbGxfY29sb3JibGluZCgpKwogIHRoZW1lX2J3KCkKYGBgCgpCeSBzdWJ0cmFjdGluZyBvdXQgdGhlIGVmZmVjdCBvZiBncm91cCBgeWAsIHdlIGFsaWduIHRoZSBkaXN0cmlidXRpb24gb2YgZGF0YSBmcm9tIGdyb3VwcyBgeGAgYW5kIGB5YCB0byBiZSBvbiB0b3Agb2YgZWFjaG90aGVyLCBhbmQgdGhlbiB0aGUgcmFuZG9tIGVmZmVjdHMgYXJlIGVzdGltYXRlZCBhcyBpZiB0aGlzIHdlcmUganVzdCBvbmUgZGlzdHJpYnV0aW9uLgoKKipCdXQqKiwgdHdvIGdyb3VwcyBvZiBzcGVha2VycyBhcmUgYWJsZSB0byBkaWZmZXIgaW4gbW9yZSB0aGFuIGp1c3QgdGhlaXIgbWVhbnMhCmZvciBleGFtcGxlLCBpdCdzIHBvc3NpYmxlIHRoYXQgdGhlc2UgdHdvIGdyb3VwcyBvZiBzcGVha2VycyBkaWZmZXIgbm90IG9ubHkgaW4gdGhlciBtZWFucywgYnV0IGFsc28gaW4gdGhlaXIgdmFyaWFuY2VzLiAKCgpgYGB7cn0Kc3BlYWtlcnMgPC0gbGV0dGVyc1sxOjZdCnNwZWFrZXJfbWVhbnNfMiA8LSBjKHJub3JtKDMsIG1lYW4gPSAwLCBzZCA9IDEpLAogICAgICAgICAgICAgICAgICAgcm5vcm0oMywgbWVhbiA9IDQsIHNkID0gMykpCiMgSGVyZSwgc3BlYWtlciBncm91cCAyIGlzIHNhbXBsZWQgZnJvbSBhIGRpc3RyaWJ1dGlvbiB3aXRoIHNkID0gMwojIGluc3RlYWQgb2Ygc2QgPSAxLCBsaWtlIGl0IHdhcyBpbiB0aGUgZmlyc3QgZXhhbXBsZS4KYGBgCmBgYHtyfQpzcGVha2VyX3NhbXBzXzIgPC0gbWFwKHNwZWFrZXJfbWVhbnNfMiwgfnJub3JtKDUwLCBtZWFuID0gLngsIHNkID0gMSkpICU+JSBzaW1wbGlmeSgpCmBgYApgYGB7ciBlY2hvICA9IEZ9CmZha2Vfc3BlYWtlcnNfMiA8LSBkYXRhLmZyYW1lKHNwZWFrZXIgPSByZXAoc3BlYWtlcnMsIGVhY2ggPSA1MCksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBncm91cCA9IHJlcChjKCJ4IiwgInkiKSwgZWFjaCA9ICg1MCAqIDMpKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG91dGNvbWUgPSBzcGVha2VyX3NhbXBzXzIpCmBgYApgYGB7ciBlY2hvICA9IEZ9CmdncGxvdChmYWtlX3NwZWFrZXJzXzIsIGFlcyhvdXRjb21lLCBmaWxsID0gZ3JvdXApKSsKICBzdGF0X2Jpbihwb3NpdGlvbiA9ICJpZGVudGl0eSIsIGFscGhhID0gMC44KSsKICBzY2FsZV9maWxsX2NvbG9yYmxpbmQoKSsKICB0aGVtZV9idygpCmBgYAoKWW91IGNhbiBmaXQgYW4gTE1FIHRvIHRoaXMgZGF0YSwgYnV0IHRoZSBmYWN0IHRoYXQgZ3JvdXAgYHlgIGhhcyBhIG11Y2ggbGFyZ2VyIHZhcmlhbmNlIHRoYW4gZ3JvdXAgYHhgIHdvbid0IGJlIGFjY291bnRlZCBmb3IuIApUaGUgdmFyaWFuY2UgZXN0aW1hdGVzIGZvciB0aGUgcmFuZG9tIGludGVyY2VwdHMgd2lsbCB3aW5kIHVwIHNwbGl0dGluZyB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBhY3R1YWwgdmFyaWFuY2VzIG9mIGdyb3VwcyBgeGAgYW5kIGB5YC4KCmBgYHtyfQpzdW1tYXJ5KGxtZXIob3V0Y29tZSB+IGdyb3VwICsgKDF8c3BlYWtlciksIGRhdGEgPSBmYWtlX3NwZWFrZXJzXzIpKQpgYGAKCiMjIyBEaWZmZXJlbnQgdmFyaWFuY2VzIHdpdGhpbiBncm91cGluZyBmYWN0b3JzCgpMZXQncyBmaXQgYSByYW5kb20gaW50ZXJjZXB0cyBvbmx5IG1vZGVsIHRvIHRoZSBgYXlUYCBkYXRhLgoKYGBge3J9CmF5VF9pbnRlcmNlcHRfbW9kIDwtIGxtZXIoRjFfbiB+ICgxfGlkc3RyaW5nKSwgZGF0YSA9IGF5VCkKc3VtbWFyeShheVRfaW50ZXJjZXB0X21vZCkKYGBgCgpMZXQncyB0aGluayBhYm91dCBob3cgdGhpcyBtb2RlbCBpcyBtb2RlbGxpbmcgdGhlIGRhdGEuCkZpcnN0LCBpdCdzIGFzc3VtaW5nIHRoYXQgdGhlIHNwZWFrZXIgcmFuZG9tIGludGVyY2VwdHMgYXJlIGJlaW5nIGRyYXduIGZyb20gYSBub3JtYWwgZGlzdHJpYnV0aW9uLgpUaGUgbWVhbiBvZiB0aGF0IGRpc3RyaWJ1dGlvbiBpcyB0aGUgZml4ZWQgZWZmZWN0cyBpbnRlcmNlcHQgZXN0aW1hdGUsIGByIHJvdW5kKGZpeGVmKGF5VF9pbnRlcmNlcHRfbW9kKVsxXSwgZGlnaXRzID0gMilgLiAKVGhlIHN0YW5kYXJkIGRldmlhdGlvbiBvZiB0aGlzIGRpc3RyaWJ1dGlvbiBpcyBlc3RpbWF0ZWQgdG8gYmUgMC4zNS4KU28sIHdlIGNvdWxkIHRyZWF0IGluZGl2aWR1YWwgc3BlYWtlcnMnIG1lYW5zIGFzIGJlaW5nIGRyYXduIGZyb20gYSBub3JtYWwgZGlzdHJpYnV0aW9uIHdpdGggJFxtdSA9IDAuNTkkIGFuZCAkXHNpZ21hPTAuMzUkLgoKIVtdKGZpZ3VyZXMvc3BlYWtlcl92YXJpYW5jZXMuc3ZnKQoKQnV0LCBldmVyeSBzcGVha2VyIGNvbnRyaWJ1dGVkIG11bHRpcGxlIGRhdGEgcG9pbnRzIHRvIHRoZSBlbnRpcmUgZGF0YSBzZXQuCkluIGFkZGl0aW9uIHRvIGEgJFxtdSQsIGV2ZXJ5IHNwZWFrZXIgbXVzdCBhbHNvIGhhdmUgYSAkXHNpZ21hJC4gCkRvZXMgdGhlIExNRSBlc3RpbWF0ZSBhICRcc2lnbWEkIGZvciBlYWNoIHNwZWFrZXI/CgpJbiBmYWN0LCB0aGUgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIHRoZSAqcmVzaWR1YWxzKiBjb3VsZCBiZSB0aG91Z2h0IG9mIGFzIHRoZSAkXHNpZ21hJCBmb3IgZWFjaCBzcGVha2VyLiAKU28sIG5vLCB0aGUgTE1FIGRvZXMgbm90IGVzdGltYXRlIGEgJFxzaWdtYSQgZm9yIGV2ZXJ5IHNwZWFrZXIuClJhdGhlciwgaXQgZXN0aW1hdGVzICoxKiAkXHNpZ21hJCBmb3IgYWxsIHNwZWFrZXJzLiAKCk5vdywgc3RyaWN0bHkgc3BlYWtpbmcsIHRoaXMgYXNzdW1wdGlvbiB0aGF0IGV2ZXJ5IHNwZWFrZXIgd2lsbCBoYXZlIHRoZSBzYW1lICRcc2lnbWEkIGlzIGFsbW9zdCBjZXJ0YWlubHkgbm90IGdvaW5nIHRvIGJlIHRydWUhIApIZXJlJ3Mgd2hhdCBpdCBsb29rcyBsaWtlIGlmIHdlIGVzdGltYXRlIHRoZSBzdGFuZGFyZCBkZXZpYXRpb24gZm9yIGV2ZXJ5IHNwZWFrZXIgd2l0aCBhIHJlYXNvbmFibGUgYW1vdW50IG9mIGRhdGEgKHRvIG1ha2Ugc3VyZSB3ZSdyZSBnZXR0aW5nIGZhaXJseSByZWxpYWJsZSBlc3RpbWF0ZXMpLCBjb21wYXJlZCB0byB0aGUgc2luZ2xlIHN0YW5kYXJkIGRldmlhdGlvbiBvZiB0aGUgcmVzaWR1YWxzIGZyb20gdGhlIExNRS4KCgoKYGBge3J9CmF5VCAlPiUKICBncm91cF9ieShpZHN0cmluZykgJT4lCiAgZmlsdGVyKG4oKSA+IDQwKSAlPiUKICBzdW1tYXJpc2UobWVhbl9GMSA9IG1lYW4oRjFfbiksCiAgICAgICAgICAgIHNkX0YxID0gc2QoRjFfbiksCiAgICAgICAgICAgIG4gPSBuKCkpICU+JQogIGdncGxvdChhZXMobWVhbl9GMSwgc2RfRjEpKSsKICAgIGdlb21faGxpbmUoeWludGVyY2VwdD0wLjUxNjAsIAogICAgICAgICAgICAgICBjb2xvciA9ICJyZWQiLAogICAgICAgICAgICAgICBzaXplID0gMykrCiAgICBnZW9tX3BvaW50KGFlcyhzaXplID0gbiksCiAgICAgICAgICAgICAgIGFscGhhID0gMC42KSsKICAgIHNjYWxlX3NpemVfYXJlYSgpKwogICAgdGhlbWVfYncoKQpgYGAKCkl0IGNvdWxkIGJlIG9mIHNvbWUgaW50ZXJlc3QgdG8gY2FwdHVyZSB0aGUgZmFjdCB0aGF0IG5vdCBhbGwgc3BlYWtlcnMgaGF2ZSB0aGUgc2FtZSAkXHNpZ21hJCwgYW5kIGl0IHdvdWxkIGRlZmluaXRlbHkgaW1wcm92ZSB0aGUgZml0IG9mIHRoZSBtb2RlbCwgYnV0IHRoaXMganVzdCBpc24ndCBzb21ldGhpbmcgdGhhdCBjYW4gYmUgY2FwdHVyZSB3aXRoIExNRXMuCgojIyMgTm9uLW5vcm1hbCBkYXRhCgpKdXN0IGxpa2Ugd2l0aCBvcmRpbmFyeSBsaW5lYXIgcmVncmVzc2lvbiwgaWYgeW91ciBvdXRjb21lIHZhcmlhYmxlIGlzIG5vbi1ub3JtYWwsIGl0J3MgdGVjaG5pY2FsbHkgZ29pbmcgdG8gdmlvbGF0ZSB0aGUgYXNzdW1wdG9ucyBvZiBub3JtYWxpdHkuCkZvciBleGFtcGxlLCBpZiBpbnN0ZWFkIG9mIGZpdHRpbmcgYSBtb2RlbCB3aGVyZSB3ZSB0cmllZCB0byBlc3RpbWF0ZSBob3cgdGhlIHByb251bmNpYXRpb24gb2YgYGF5VGAgY2hhbmdlZCwgd2UgdHJpZWQgdG8gZml0IGEgbW9kZWwgdG8gc2VlIHdoZXRoZXIgdGhlICpkdXJhdGlvbiogb2YgL2F5VC8gaGFzIGNoYW5nZWQgb3ZlciB0aW1lPwoKYGBge3J9CnN1bW1hcnkobG1lcihkdXIgfiBkZWNhZGUgKyAoMXxpZHN0cmluZyksIGRhdGEgPSBheVQpKQpgYGAKCkhlcmUsIHdlIGdldCBhIGZhaXJseSByZWxpYWJsZSBlZmZlY3QgKGxvb2tzIGxpa2UgL2F5VC8gaGFzIGdvdHRlbiBzaG9ydGVyIG92ZXIgdGltZSksIGJ1dCByZWNhbGwgdGhhdCB0aGUgZGlzdHJpYnV0aW9uIG9mIGR1cmF0aW9uIGxvb2tzIGxpa2UgdGhpcyEKCjxkaXYgY2xhc3MgPSAiaGFsZi1pbWciPgpgYGB7ciBlY2hvID0gRiwgZmlnLndpZHRoPTIsIGZpZy5oZWlnaHQgPSAyfQpnZ3Bsb3QoYXlULCBhZXMoZHVyKSkrCiAgc3RhdF9kZW5zaXR5KCkrCiAgdGhlbWVfYncoKQpgYGAKPC9kaXY+CgoKSXQncyBub3QgbGlrZWx5IHRoYXQgZWl0aGVyIHRoZSBieS1zcGVha2VyIHJhbmRvbSBpbnRlcmNlcHRzIG5vciB0aGUgcmVzaWR1YWxzIHdpbGwgaGF2ZSBhIG5vcm1hbCBkaXN0cmlidXRpb24uIApUaGlzIGlzIGFjdHVhbGx5IG9uZSBzY2VuYXJpbyB5b3UgY2FuIGdldCBhcm91bmQgYmVjYXVzZSBpdCdzIHByaW5jaXBsZWQgZW5vdWdoIHRvIGxvZy1zY2FsZSB0aGUgZHVyYXRpb24gb3V0Y29tZSwgYnV0IHRoZXJlIHdvbid0IGFsd2F5cyBiZSBhIHByaW5jaXBsZWQgdHJhbnNmb3JtIG9mIHNrZXdlZCBkYXRhIGxpa2UgdGhpcy4KCgoKCgojIEZpeGVkIEVmZmVjdCBvciBSYW5kb20gRWZmZWN0PwoKQmVmb3JlIGV2ZW4gZ2V0dGluZyB0byBob3cgdG8gc3BlY2lmeSBhIHJhbmRvbSBlZmZlY3RzIHN0cnVjdHVyZSwgcGVvcGxlIG9mdGVuIGFsc28gd29uZGVyIHdoaWNoIHByZWRpY3RvcnMgc2hvdWxkIGJlIGluY2x1ZGVkIGFzIGEgZml4ZWQgZWZmZWN0LCBhbmQgd2hpY2ggc2hvdWxkIGJlIGluY2x1ZGVkIGFzIGEgcmFuZG9tIGVmZmVjdC4gCkZvciBleGFtcGxlLCB3aGVuIGxvb2tpbmcgYXQgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIGR1cmF0aW9uIGFuZCBGMSBmb3IgL2F5VC8sIGxldCdzIHN1cHBvc2Ugd2UgaGFkIHRoZSBmb2xsb3dpbmcgc3VzcGljaW9uOgoKPiBJIHRoaW5rIHRoYXQgZGlmZmVyZW50IHNwZWFrZXJzIG1pZ2h0IGhhdmUgZGlmZmVyZW50IHNsb3BlcyBmb3IgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIGR1cmF0aW9uIGFuZCBGMS4KClRoZXJlIGFyZSB0d28gZGlmZmVyZW50IHdheXMgd2UgY291bGQgYXBwcm9hY2ggdGhpcwoKMS4gRml0IGEgbGluZWFyIG1vZGVsIHdpdGggYW4gaW50ZXJhY3Rpb24gYmV0d2VlbiBkdXJhdGlvbiBhbmQgc3BlYWtlci4KMi4gRml0IGEgbGluZWFyIG1peGVkIGVmZmVjdHMgbW9kZWwgd2l0aCBhIHJhbmRvbSBzbG9wZSBvZiBkdXJhdGlvbiBieSBzcGVha2VyLgoKTGV0J3MganVzdCBsb29rIGF0IHRoZSBmaXJzdCBvcHRpb24gYnJpZWZseToKCmBgYHtyfQpheVRfbG1faW50ZXJhY3Rpb24gPC0gbG0oRjFfbiB+IGR1cl9jICogaWRzdHJpbmcsIGRhdGEgPSBheVQpCmBgYAoKYGBge3J9CnN1bW1hcnkoYXlUX2xtX2ludGVyYWN0aW9uKQpgYGAKCkkndmUgYWN0dWFsbHkgbGVmdCB0aGUgc3VtbWFyeSBvZiB0aGlzIG1vZGVsIG91dCBvZiB0aGUgbm90ZXMsIGJlY2F1c2UgaXQncyB0b28gbG9uZyB0byBiZSBvZiBhbnkgdXNlLiAKV2hhdCB3ZSB3aW5kIHVwIHdpdGggaXMgYW4gaW50ZXJjZXB0IGFuZCBzbG9wZSBlc3RpbWF0ZSBmb3Igc29tZSByZWZlcmVuY2Ugc3BlYWtlciwgYW5kIHRoZW4gdGhlIGRpZmZlcmVuY2Ugb2YgZXZlcnlvdGhlciBzcGVha2VycycgaW50ZXJjZXB0cyBhbmQgc2xvcGVzIGZyb20gdGhhdCByZWZlcmVuY2Ugc3BlYWtlci4KVGhlIHdlaXJkbmVzcyBvZiBjaG9vc2luZyBhIHJlZmVyZW5jZSBsZXZlbCBzcGVha2VyIGNhbiBiZSBhbWVsaW9ydGVkIGJ5IHVzaW5nIHN1bS1jb250cmFzdHMsIGJ1dCB0aGF0IGRvZXNuJ3QgYWRkcmVzcyBhIG1vcmUgc2VyaW91cyBpc3N1ZTogVGhpcyBhcHByb2FjaCB3aWxsIHdpbmQgdXAgZ2l2aW5nIHVzIGVzdGltYXRlcyAqaWRlbnRpY2FsKiB0byB0aGUgbm8tcG9vbGluZyBtZXRob2QuCgpFYWNoIHNwZWFrZXIncyBjb2VmZmljaWVudCBhbmQgc2xvcGUgaW50ZXJhY3Rpb24gaXMgZXN0aW1hdGVkIHdpdGhvdXQgcmVzcGVjdCB0byB0aGUgZXN0aW1hdGVzIG9mIGFueSBvdGhlciBzcGVha2VyLiAKQW5kIHRoYXQgY29tZXMgd2l0aCBhbGwgb2YgdGhlIHNhbWUgc2hvcnRjb21pbmdzIHdlIGRlYWx0IHdpdGggaW4gdGhlIG5vLXBvb2xpbmcgbW9kZWwuCgoKT24gdGhlIG90aGVyIGhhbmQsIGxldCdzIHJldHVybiB0byB0aGUgYnJvYWRlciBgYXlgIGRhdGEgc2V0LgpSZWNhbGwgd2UndmUgYmVlbiBsb29raW5nIGF0IC9heS8gaW4gc3RyaWN0bHkgcHJlLXZvaWNlbGVzcyBjb250ZXh0LiAKQnV0IGluIHRoZSBwcmUtdm9pY2VkIGNvbnRleHQsIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiBGMSBhbmQgdGhlIHNwZWFrZXJzJyBkYXRlIG9mIGJpcnRoIGxvb2tzIGFsb3QgZGlmZmVyZW50LgoKYGBge3J9CmF5IDwtICBheSAlPiUgbXV0YXRlKGRvYiA9IHllYXItYWdlLCAKICAgICAgICAgICAgICAgICAgICBkZWNhZGUgPSAoZG9iIC0gMTkwMCkvMTApCmF5ICU+JQogIGdyb3VwX2J5KGlkc3RyaW5nLCBkb2IsIHBsdF92Y2xhc3MpJT4lCiAgc3VtbWFyaXNlKEYxX24gPSBtZWFuKEYxX24pKSAlPiUKICBnZ3Bsb3QoYWVzKGRvYiwgRjFfbiwgY29sb3IgPSBwbHRfdmNsYXNzKSkrCiAgICBnZW9tX3BvaW50KCkrCiAgICBzY2FsZV9jb2xvcl9jb2xvcmJsaW5kKCkrCiAgICB0aGVtZV9idygpCmBgYAoKTG9va2luZyBhdCB0aGUgZ3JhcGgsIHdlIG1pZ2h0IHNheToKCj4gSSBzdXNwZWN0IHRoYXQgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIEYxIGFuZCBkZWNhZGUgd2lsbCBkaWZmZXIgYmV0d2VlbiBwcmUtdm9pY2VkIGFuZCBwcmUtdm9pY2VsZXNzIGNvbnRleHRzLgoKVGhlIHdheSB0byBtb2RlbCB0aGlzIHdvdWxkIGJlIHRvIGVpdGhlcjoKCjEuIEZpdCBhIG1vZGVsIHdpdGggZml4ZWQgZWZmZWN0cyBpbnRlcmFjdGlvbiBiZXR3ZWVuZCBkZWNhZGUgYW5kIHZvaWNpbmcuCjIuIEZpdCBhIG1vZGVsIHdpdGggcmFuZG9tIHNsb3BlIG9mIGRlY2FkZSBieSB2b2ljaW5nLgoKSnVzdCBmb3IgaWxsdXN0cmF0aW9uLCBsZXQncyBsb29rIGF0IGEgbW9kZWwgb2YgdGhpcyBzZWNvbmQgYXBwcm9hY2g6CgpgYGB7cn0KYXlfbG1lcl9zbG9wZSA8LSBsbWVyKEYxX24gfiBkZWNhZGUgKyAoMXxpZHN0cmluZykgKyAoMSArIGRlY2FkZSB8IHBsdF92Y2xhc3MpLAogICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGF5KQoKc3VtbWFyeShheV9sbWVyX3Nsb3BlKQpgYGAKCldlIGdldCBvdXQgb25lIGZpeGVkIGVmZmVjdHMgaW50ZXJjZXB0IGFuZCAgc2xvcGUsIGFuZCBhbiBlc3RpbWF0ZSBvZiB0aGUgdmFyaWFuY2Ugb2YgdGhlIHJhbmRvbSBzbG9wZXMgYnkgdm9pY2luZyBjb250ZXh0LgpXZSBjYW4gYWxzbyBsb29rIGF0IHRoZSByYW5kb20gZWZmZWN0cyBwcmVkaWN0aW9uczoKCmBgYHtyfQpyYW5lZihheV9sbWVyX3Nsb3BlKSRwbHRfdmNsYXNzCmBgYAoKT25lIHRoaW5nIHRoYXQgaXNuJ3Qgc28gZ3JlYXQgYWJvdXQgdGhpcyByZXN1bHQgaXMgdGhhdCBzaHJpbmthZ2Ugd2lsbCBhaHZlIGFwcGxpZWQgdG8gdGhlc2UgcmFuZG9tIHNsb3BlcyEKVGhhdCBpcywgdGhlIHByZWRpY3RlZCBzbG9wZXMgZm9yIHByZS12b2ljZWQgYW5kIHByZS12b2ljZWxlc3MgL2F5LyBhcmUgZ29pbmcgdG8gYmUgbGVzcyBkaWZmZXJlbnQgaGVyZSB0aGFuIHRoZXkgd291bGQgYmUgYXMgZml4ZWQgZWZmZWN0cy4KCi0tLS0tLS0KCldlIGNhbiBzdW0gdXAgdGhlIG1vdGl2YXRpb24gZm9yIGluY2x1ZGluZyBhIGZhY3RvciBhcyBhIGZpeGVkIG9yIHJhbmRvbSBlZmZlY3QgYmFzZWQgb24gd2hldGhlciBvciBub3Qgd2Ugd2FudCBzaHJpbmthZ2UsIG9yIHBhcnRpYWwgcG9vbGluZywgdG8gYXBwbHkgYmV0d2VlbiBsZXZlbHMgb2YgdGhlIGZhY3Rvci4gCk9yLCBhbm90aGVyIHdheSBvZiBwdXR0aW5nIGl0IGlzIHdoZXRoZXIgb3Igbm90IHdlIHdhbnQgdG8gdHJlYXQgdGhlIHBhcmFtZXRlciB2YWx1ZXMgZm9yIGVhY2ggbGV2ZWwgb2YgdGhlIGZhY3RvciBhcyBiZWluZyBkcmF3biBmcm9tIGEgcG9wdWxhdGlvbi1sZXZlbCBkaXN0cmlidXRpb24uIApUaGF0IGlzLCB0aGUgZGVjaXNpb24gaXMgKmFsd2F5cyogZ29pbmcgdG8gYmUgZ3VpZGVkIGJ5IHlvdXIgY29uY2VwdHVhbGl6YXRpb24gb2YgdGhlIGRhdGEsIGV2ZW4gdGhvdWdoIHRoZXJlIGFyZSB1c3VhbGx5IGRlY2lzaW9ucyB0aGF0IGNhbiBiZSBtYWRlIGJ5IGNvbnZlbnRpb24uIEFsbW9zdCBhbHdheXMsIHRoZSBmb2xsb3dpbmcgY2FuIGJlIHRyZWF0ZWQgYXMgcmFuZG9tIGVmZmVjdHMKCi0gaW5kaXZpZHVhbHMgKHNwZWFrZXJzLCBzdWJqZWN0cywgdGV4dHMsIGF1dGhvcnMsIGV0Yy4pCi0gaXRlbXMgKHdvcmRzLCBzcGVjaWZpYyBzdGltdWxpLCBldGMuKQoKQW5kIG1vc3QgdGhpbmdzIHRoYXQgeW91IHdhbnQgdG8gbG9vayBhdCBmb3IgYSBnZW5lcmFsaXphYmxlIGVmZmVjdCBjYW4gYmUgZml4ZWQgZWZmZWN0cy4KCiMjICJOdWlzYW5jZSB2YXJpYWJsZXMiCgpBbm90aGVyIHdheSB0byB0aGluayBhYm91dCByYW5kb20gZWZmZWN0cyBpcyBhcyAibnVpc2FuY2UgdmFyaWFibGVzIi4KV2hlbiBtb2RlbGxpbmcgc29tZXRoaW5nIGFzIGNvbXBsZXggYXMgd2h5IHByb251bmNpYXRpb25zIGNoYW5nZSwgdGhlcmUgYXJlIGFsd2F5cyBnb2luZyB0byBiZSBmYWN0b3JzIG1pc3NpbmcgZnJvbSBvdXIgbW9kZWxzIGZvciBhIHZhcmlldHkgb2YgcmVhc29ucy4KCi0gdGhlcmUgYXJlIHRoaW5ncyB3ZSBrbm93IG1hdHRlciwgYnV0IGZhaWxlZCB0byBtZWFzdXJlIHRoZW0gZm9yIHNvbWUgcmVhc29uLgotIHRoZXJlIGFyZSB0aGluZ3Mgd2Uga25vdyBtYXR0ZXIsIGJ1dCB3ZSBkb24ndCBrbm93ICpob3cqIHRvIG1lYXN1cmUgdGhlbS4KLSB0aGVyZSBhcmUgdGhpbmdzIHRoYXQgbWF5IG5vdCBiZSAqbWVhc3VyZWFibGUqLgotIHRoZXJlIGFyZSB0aGluZ3Mgd2UgZG9uJ3QgcmVhbGl6ZSBtYXR0ZXIuCgpJZiB3ZSBkb24ndCBkbyBvdXIgYmVzdCB0byBhY2NvdW50IGZvciB0aGVzZSAidW5tb2RlbGxhYmxlcyIsIHRoZXkgY291bGQgdmVyeSB3ZWxsIG1lc3Mgd2l0aCBvdXIgZml4ZWQgZWZmZWN0cyBlc3RpbWF0ZXMuIApXaGlsZSB0aGV5IGFyZSBub3QgYSBwZXJmZWN0IHNvbHV0aW9uLCByYW5kb20gZWZmZWN0cyBjYW4gZ28gc29tZSB3YXkgdG93YXJkcyBzb2FraW5nIHVwIHNvbWUgb2YgdGhlc2UgdW5tb2RlbGFibGUgdmFyaWFibGVzLgoKT24gdGhlIGZsaXAgc2lkZSwgaWYgeW91IGhhdmUgc29tZXRoaW5nIHlvdSAqa25vdyogaGFzIGFuIGVmZmVjdCwgaW5jbHVkZSBpdCBpbiB0aGUgbW9kZWwuIApGb3IgZXhhbXBsZSwgd2Uga25vdyB0aGF0IGdlbmRlciBoYXMgYSBiaWcgZWZmZWN0IG9uIG1hbnkgY2hhbmdlcyBpbiBwcm9ncmVzcy4KWW91IHNob3VsZCwgdGhlcmVmb3JlLCBpbmNsdWRlIGl0IGluIHRoZSBtb2RlbCBpbnN0ZWFkIG9mIGhvcGluZyB0aGF0IHRoZSByYW5kb20gZWZmZWN0cyB3aWxsIGFjY291bnQgZm9yIGl0LgpUaGF0J3MgYmVjYXVzZSBpZiB0aGVyZSBpcyBhIGJpZyBiZXR3ZWVuLXN1YmplY3QgdmFyaWFibGUgdGhhdCBhZmZlY3RzIHRoZSBvdXRjb21lIHZhcmlhYmxlLCB0aGUgYXNzdW1wdGlvbiB0aGF0IHNwZWFrZXIgcmFuZG9tIGludGVyY2VwdHMgJiBzbG9wZXMgYXJlIGRyYXduIGZyb20gYSBzaW5nbGUgbm9ybWFsIGRpc3RyaWJ1dGlvbiBpcyB2aW9sYXRlZC4KCgojIFBpdGZhbGxzIGluIG1peGVkIGVmZmVjdHMgbW9kZWxsaW5nCgpUaGUgYmlnZ2VzdCBpc3N1ZSBwZW9wbGUgZmFjZSB3aGVuIGZpdHRpbmcgYSBtaXhlZCBlZmZlY3RzIG1vZGVsIGlzIHRoYXQgaXQgbWF5ICJmYWlsIHRvIGNvbnZlcmdlIi4gVGhpcyBpcyBiZWNhdXNlIHRoZXNlIG1vZGVscyBhcmVuJ3QgJ3NvbHZlZCcgaW4gdGhlIHNhbWUgd2F5IGFzIHRoZSByZWd1bGFyIGxpbmVhciByZWdyZXNzaW9uIGlzLgpCZWNhdXNlIG9mIHRoZWlyIG1vcmUgY29tcGxpY2F0ZWQgbmF0dXJlLCB0aGVzZSBtb2RlbHMgYXJlIHVzdWFsbHkgZXRpbWF0ZWQgdmlhIGEgcHJvY2VzcyBvZiBpdGVyYXRpdmUgb3B0aW1pemF0aW9uLgoKSXQgaXMgdmVyeSBpbXBvcnRhbnQgdG8gb25seSBiYXNlIHlvdXIgaW5mZXJlbmNlIG9uIGNvbnZlcmdlZCBtb2RlbHMuIFRoZXJlIGFyZSBhIGhhbmRmdWwgIG9mIHR3ZWFrcyB0aGF0IG1pZ2h0IGhlbHAgaW4geW91ciBtb2RlbHMgY29udmVyZ2luZy4KCi0gY2VudGVyIGFuZCBzY2FsZSBhbGwgY29udGludW91cyBwcmVkaWN0b3JzLgotIGlmIHBvc3NpYmxlLCBjaG9vc2UgYSBkaWZmZXJlbnQgZmFjdG9yIGxldmVsIHRoYXQgaGFzIG1vcmUgZGF0YSBhcyB5b3VyIHJlZmVyZW5jZSBsZXZlbC4KLSByZW1vdmUgdGhlIGNvcnJlbGF0aW9uIGNvbXBvbmVudCBmcm9tIHlvdXIgbW9kZWwKCgpUaGUgaW1wb3JhbmNlIG9mIGNlbnRlcmluZyBhbmQgc2NhbGluZyB5b3VyIGNvbnRpbnVvdXMgcHJlZGljdG9ycyBhcmUgdmVyeSBpbXBvcnRhbnQgaGVyZS4gCkxldCdzIGxvb2sgYXQgYSByZWxhdGl2ZWx5IHNpbXBsZSBtb2RlbCB3aGljaCBpcyB0cnlpbmcgdG8gcHJlZGljdCBGMSBieSBkYXRlIG9mIGJpcnRoLCB3aXRoIGEgcmFuZG9tIGludGVyY2VwdCBieSBzcGVha2VyLCBhbmQgYSByYW5kb20gaW50ZXJjZXB0IGFuZCBzbG9wZSBieSB3b3JkLgoKYGBge3J9CmF5X2ZhaWwgPC0gbG1lcihGMV9uIH4gZG9iICsgKDEgfCBpZHN0cmluZykgKyAoMSArIGRvYiB8IHdvcmQpLAogICAgICAgICAgICAgICAgZGF0YSA9IGF5VCkKYGBgCmBgYHtyfQpzdW1tYXJ5KGF5X2ZhaWwpCmBgYAoKVGhpcyBpcyAqYmFkKi4KQnV0IHJlbWVtYmVyLCB0aGUgYGRvYmAgdmFyaWFibGUgaXMgbG9va2luZyBhdCB0aGUgZGF0YSBsaWtlIHRoaXM6Cgo8ZGl2IGNsYXNzID0gImhhbGYtaW1nIj4KYGBge3IsIGVjaG8gPSBGfQpnZ3Bsb3QoYXlULCBhZXMoZG9iLCBGMV9uKSkrCiAgICBnZW9tX3BvaW50KCkrCiAgICBnZW9tX2FibGluZShhZXMoaW50ZXJjZXB0ID0gMjMuNDY1MjE2MCwKICAgICAgICAgICAgICAgICAgICBzbG9wZSA9IC0wLjAxMTY4MjMpLAogICAgICAgICAgICAgICAgY29sb3IgPSAicmVkIikrCiAgICB4bGltKDAsIDE5OTgpKwogICAgeWxpbSgtMiwgMjMpKwogICAgdGhlbWVfYncoKQpgYGAKPC9kaXY+CgpUaGlzIGNhbiBwb3NlIGEgY2hhbGxlbmdlIGZvciB0cnlpbmcgdG8gZXN0aW1hdGUgcmFuZG9tIGludGVyY2VwdHMgYW5kIHNsb3BlcyEgSWYgd2UgcmVmaXQgdGhlIG1vZGVsLCB0aGlzIHRpbWUgd2l0aCB0aGUgY2VudGVyZWQgYW5kIHNjYWxlZCBgZGVjYWRlYCBwcmVkaWN0b3IsIGl0J2xsIGNvbnZlcmdlLgoKYGBge3J9CmF5X3dpbiA8LSBsbWVyKEYxX24gfiBkZWNhZGUgKyAoMSB8IGlkc3RyaW5nKSArICgxICsgZGVjYWRlIHwgd29yZCksCiAgICAgICAgICAgICAgICBkYXRhID0gYXlUKQpgYGAKYGBge3J9CnN1bW1hcnkoYXlfd2luKQpgYGAKCkJ1dCBzb21ldGltZXMsIG5vIG1hdHRlciBob3cgeW91IHNjYWxlIG9yIGNlbnRlciB5b3VyIHByZWRpY3RvcnMsIHRoZSByYW5kb20gZWZmZWN0cyBzdHJ1Y3R1cmUgbWlnaHQgYmUgdG9vIGNvbXBsZXggZ2l2ZW4gdGhlIGRhdGEuCkkgdHJpZWQgZml0dGluZyBhIG1vZGVsIGxpa2UgdGhpcyB3aXRoIHRoZSAvYXlULyBkYXRhLCBidXQgaXQgYWN0dWFsbHkgY29udmVyZ2VkOgoKCmBgYHtyfQpheV9iaWcgPC0gbG1lcihGMV9uIH4gZGVjYWRlICogZHVyX2MgKyAKICAgICAgICAgICAgICAgICAgKDEgKyBkdXJfYyB8IGlkc3RyaW5nKSArIAogICAgICAgICAgICAgICAgICAoMSArIGRlY2FkZSAqIGR1cl9jIHwgd29yZCksCiAgICAgICAgICAgICAgIGRhdGEgPSBheVQpCmBgYAoKYGBge3J9CnN1bW1hcnkoYXlfYmlnKQpgYGAKCkJ1dCBvZnRlbiwgYSBtb2RlbCBsaWtlIHRoaXMgd2lsbCBmYWlsIHRvIGNvbnZlcmdlLCBhbmQgaXQncyBlYXN5IGVub3VnaCB0byB1bmRlcnN0YW5kIHdoeS4KRm9yIG1hbnkgbGV2ZWxzIG9mIGB3b3JkYCwgdGhlcmUgaXMgb25seSAxIHRva2VuLCBidXQgdGhlIG1vZGVsIGlzIHRyeWluZyB0byBtb2RlbCBhIHJhbmRvbSBpbnRlcmNlcHQsIHNsb3BlIGZvciBkZWNhZGUgYW5kIGR1cmF0aW9uLCB0aGVpciBpbnRlcmFjdGlvbiwgYW5kIHRoZWlyIGNvdmFyaWFuY2VzLCBmb3IgKmV2ZXJ5KiBsZXZlbCBvZiBgd29yZGAuCk9uZSB0aGluZyB0aGF0IGNvdWxkIG1ha2UgYSBiaWcgbW9kZWwgbGlrZSB0aGlzIHRyYWN0aWJsZSBpcyB0byBlbGltaW5hdGUgdGhlIGNvdmFyaWFuY2Ugc3RydWN0dXJlIHVzaW5nIHRoZSBgKHBhcmFtZXRlciB8fCBncm91cClgIHN5bnRheAoKYGBge3J9CmF5X2JpZ2lzaCA8LSBsbWVyKEYxX24gfiBkZWNhZGUgKiBkdXJfYyArIAogICAgICAgICAgICAgICAgICAoMSArIGR1cl9jIHwgaWRzdHJpbmcpICsgCiAgICAgICAgICAgICAgICAgICgxICsgZGVjYWRlICogZHVyX2MgfHwgd29yZCksCiAgICAgICAgICAgICAgIGRhdGEgPSBheVQpCmBgYApgYGB7cn0Kc3VtbWFyeShheV9iaWdpc2gpCmBgYAoKCldlJ2xsIHRhbGsgbW9yZSBhYm91dCBpc3N1ZXMgbGlrZSB0aGlzIG1vcmUgbmV4dCB3ZWVrIHdoZW4gd2UgdGFsayBhYm91dCBidWlsaW5nIHVwIG1vZGVscy4KCgojIyBUeXBlIEkgYW5kIFR5cGUgSUkgRXJyb3IKClRoZXNlIGFyZSBwb29ybHkgbmFtZWQsIGFuZCBJIHRoaW5rIGZldyBwZW9wbGUgcmVtZW1iZXIgd2hpY2ggaXMgd2hpY2ggY29ycmVjdGx5IHRoZSBmaXJzdCB0aW1lLiBUaGVzZSBhcmUgZXJyb3JzIHJlbGF0ZWQgdG8gdGhlIHVzZSBvZiBwLXZhbHVlcy4gRmlyc3QgaXQncyBpbXBvcnRhbnQgdG8gZGVmaW5lIHdoYXQgdGhlICoqTnVsbCBIeXBvdGhlc2lzKiogaXMuCgotICoqVGhlIE51bGwgSHlwb3RoZXNpcyBpcyB0aGUgRHVsbCBIeXBvdGhlc2lzOioqIFRoZSBOdWxsIEh5cG90aGVzaXMgaXMgdGhhdCB0aGVyZSBpcyBubyByZWxhdGlvbnNoaXAgYmV0d2VlbiB0aGUgb3V0Y29tZSBhbmQgcHJlZGljdG9ycy4gSXQgaXMgKm5vdCogd2hhdCB5b3UgdGhpbmsgdGhlIG1vc3QgbGlrZWx5IHJlbGF0aW9uc2hpcCBpcy4KCldpdGggdGhhdCBpbiBtaW5kLCB3ZSBjYW4gZGVmaW5lIFR5cGUgSSBhbmQgVHlwZSBJSSBlcnJvcnMuCgotICoqVHlwZSBJKiogLSBZb3UgcmVqZWN0IHRoZSBudWxsIGh5cG90aGVzaXMgKGJlY2F1c2UgcCA8IDAuMDUpLCBidXQgaW4gcmVhbGl0eSB0aGVyZSAqaXNuJ3QqIGEgcmVsYXRpb25zaGlwLiBUaGF0IGlzLCB5b3Ugc2F5IHRoZXJlIGlzIGEgcmVsaWFibGUgZWZmZWN0IHdoZW4gdGhlcmUgaXNuJ3Qgb25lLgotICoqVHlwZSBJSSoqIC0gWW91IGZhaWwgdG8gcmVqZWN0IHRoZSBudWxsIGh5cG90ZWhzaXMgd2hlbiB0aGVyZSAqaXMqIGFuIHJlYWwgZWZmZWN0LiAKCk9yIHRvIHB1dCBpdCBhbm90aGVyIHdheSwgd2hlbiB5b3Ugc2VlIGEgc3RhdGlzdGljYWwgcmVzdWx0IHRoYXQgaXMgc2lnbmlmaWNhbnQgKHAgPCAwLjA1KSwgYnV0IHlvdSBkb24ndCB0aGluayBpdCBjb3VsZCBwb3NzaWJseSBiZSByZWFsLCB5b3UgdGhpbmsgYSBUeXBlIEkgZXJyb3IgaGFzIG9jY3VycmVkLgoKQW5kIHdoZW4geW91IGdldCBwID4gMC4wNSwgYnV0IHlvdSByZWFsbHkgdGhpbmsgdGhlcmUncyBzb21ldGhpbmcgdGhlcmUgc28geW91IHB1dCAidHJlbmRpbmcgdG93YXJkcyBzaWduaWZpY2FuY2UiIGluIHlvdXIgcGFwZXIsIHlvdSB0aGluayBhIFR5cGUgSUkgZXJyb3IgaGFzIG9jY3VycmVkLgoKVGhlIGJlc3Qgd2F5IHRvIGFtZWxpb3JhdGUgVHlwZSBJIGFuZCBUeXBlIElJIGVycm9ycyBpcyB0byBlaXRoZXIgY29sbGVjdCBtb3JlIGRhdGEsIG9yIHRvIG1vdmUgdG8gQmF5ZXNpYW4gU3RhdGlzdGljcywgd2hpY2ggd2Ugd29uJ3QgYmUgYWJsZSB0byBjb3ZlciBpbiB0aGlzIGNvdXJzZSwgW2J1dCBJIHJlY29tbWVuZCB0aGlzIGJvb2tdKGh0dHBzOi8vc2l0ZXMuZ29vZ2xlLmNvbS9zaXRlL2RvaW5nYmF5ZXNpYW5kYXRhYW5hbHlzaXMvKS4KCgoKIAogCiMjIFR5cGUgTSBhbmQgVHlwZSBTIGVycm9ycwoKVGhlc2UgYXJlIGVycm9yIHR5cGVzIFtkaXNjdXNzZWQgYnkgR2VsbWFuIGFuZCBjb2xsZWFndWVzXShodHRwOi8vd3d3LnN0YXQuY29sdW1iaWEuZWR1L35nZWxtYW4vcmVzZWFyY2gvcHVibGlzaGVkL1BQUzU1MTY0Ml9SRVYyLnBkZikgKGFuZCBhcmUgYmV0dGVyIG5hbWVkKS4KCiAtICoqVHlwZSBNIEVycm9ycyoqIGFyZSBncm9zcyBvdmVyLWVzdGltYXRpb24gb2YgdGhlICptYWduaXR1ZGUqIG9mIGFuIGVmZmVjdCBzaXplLgogLSAqKlR5cGUgUyBFcnJvcnMqKiBnZXQgdGhlICpzaWduKiB3cm9uZyBvbiBhbiBlc3RpbWF0ZS4gVGhhdCBpcywgdGhleSBwcmVkaWN0IGFuIGVmZmVjdCBpbiB0aGUgb3Bwb3NpdGUgZGlyZWN0aW9uIG9mIHdoYXQgdGhlIHRydWUgZWZmZWN0IGlzLgogClR5cGUgTSBtaWdodCBub3Qgc291bmQgbGlrZSBhIHZlcnkgYmlnIHByb2JsZW0sIGFuZCBUeXBlIFMgbWlnaHQgc291bmQgYSBiaXQgb3V0bGFuZGlzaCBidXQgdGhlc2UgY2FuIGJvdGggaGFwcGVuLCBlc3BlY2lhbGx5IHdoZW4gd2UgYXJlIFtleHBsb3JpbmcgZm9yIHNtYWxsIGVmZmVjdHMgYW5kIGZpbHRlcmluZyByZXN1bHRzIGJ5IHRoZWlyIHAtdmFsdWVzXShodHRwOi8vcnB1YnMuY29tL0pvRnJod2xkLzI5MTczNCkuIEdlbG1hbiBoYXMgcHJvcG9zZWQgdGhhdCBbVHlwZSBNIGVycm9ycyBhcmUgYSBsaWtlbHkgY3VscHJpdCBpbiB0aGUgc28tY2FsbGVkICJkZWNsaW5lIGVmZmVjdCJdKGh0dHA6Ly9hbmRyZXdnZWxtYW4uY29tLzIwMTAvMTIvMTMvdGhlX3RydXRoX3dlYXJzLyksIHdoaWNoIGlzIGEgcGhlbm9tZW5vbiBzb21lIGhhdmUgZm91bmQgdGhhdCBsYXJnZSBhbmQgc2lnbmlmaWNhbnQgZWZmZWN0cyBncmFkdWFsbHkgZ2V0IHNtYWxsZXIgYWZ0ZXIgdGhlaXIgb3JpZ2luYWwgcmVwb3J0cy4KCkhlcmUgYXJlIHNvbWUgd2F5cyB0byBndWFyZCB5b3VzZWxmIGFnYWluc3QgVHlwZSBNIGFuZCBUeXBlIFMgZXJyb3JzLgoKLSBUcnkgdG8gcmVhc29uIGhvdyBiaWcgdGhlIGVmZmVjdCBpcyAqbGlrZWx5KiB0byBiZS4KLSBUcnkgdG8gY29sbGVjdCBvZiBob3cgb3RoZXIgZmFjdG9ycyBlZmZlY3QgdGhlIG91dGNvbWUgeW91J3JlIGludmVzdGlnYXRpbmcsIGFuZCBob3cgYmlnIHRob3NlIGVmZmVjdHMgYXJlLgotIENvbW1pdCB0byB0aGUgKmRpcmVjdGlvbiogb2YgdGhlIGVmZmVjdC4KClR5cGUgTSBhbmQgVHlwZSBTIGVycm9ycyBhcmUgb25seSBnb2luZyB0byBncm93IGluIHNldmVyaXR5IGFzIFt0aGUgc2l6ZSBvZiBkYXRhIHNldHMgZ3Jvd10oaHR0cDovL2pvZnJod2xkLmdpdGh1Yi5pby9wYXBlcnMvcGxjMzlfMjAxNS8jLykuCgoKCiMjIENvbGxpbmVhcml0eQoKIkNvbGxpbmVhcml0eSIgaXMgd2hlbiB0d28gcG90ZW50aWFsIHByZWRpY3RvcnMgYXJlIGhpZ2hseSBjb3JyZWxhdGVkIHdpdGggZWFjaCBvdGhlci4gV2hlbiB0d28gY29ycmVsYXRlZCBwcmVkaWN0b3JzIGFyZSBpbmNsdWRlZCBpbiB0aGUgc2FtZSByZWdyZXNzaW9uLCB0aGUgcmVzdWx0cyBjYW4gYmUgKndvbmt5KiBhdCBiZXN0LiBbR29ybWFuICgyMDEwKV0oaHR0cDovL3JlcG9zaXRvcnkudXBlbm4uZWR1L2NnaS92aWV3Y29udGVudC5jZ2k/YXJ0aWNsZT0xMTQ2JmNvbnRleHQ9cHdwbCkgdGFsa3MgYWJvdXQgdGhpcyBhdCBsZW5ndGggd2l0aCByZXNwZWN0IHRvIHNvY2lvZWNvbm9taWMgbWVhc3VyZXMgaW4gdGhlIExhbmd1YWdlIENoYW5nZSBhbmQgVmFyaWF0aW9uIHN0dWR5LiBGb3IgZXhhbXBsZSwgc3BlYWtlcnMnIE9jY3VwYXRpb24gd2FzIGhpZ2hseSBjb3JyZWxhdGVkIHdpdGggdGhlIHZhbHVlIG9mIHRoZWlyIFJlc2lkZW5jZS4gWW91ICpzaG91bGRuJ3QqIGluY2x1ZGUgdGhlc2UgYm90aCwgdW50cmVhdGVkLCBpbnRvIHlvdXIgbW9kZWxzLiAKCgpUaGVyZSBhcmUgYXQgbGVhc3QgdGhyZWUgZGlmZmVyZW50IHN0cmF0ZWdpZXMgZm9yIGRlYWxpbmcgd2l0aCB0aGlzIGlzc3VlCgoxLiBJZiB5b3UgKm11c3QqIG1vZGVsIHRoZXNlIGZhY3RvcnMgaW5kZXBlbmRlbnRseSwgcmVzaWR1YWxpemUgKEdvcm1hbiwgMjAxMCkKMi4gKk9yKiwganVzdCBwaWNrIHRoZSBvbmUgdGhhdCBpcyB0aGVvcmV0aWNhbGx5IG1vdGl2YXRlZCAoRnJ1ZWh3YWxkLCAyMDE2KS4KMy4gKk9yKiBnaXZlIHVwIG9uIHRyZWF0aW5nIHlvdXIgbWVhc3VyZXMgYXMgaGF2aW5nIGluZGVwZW5kZW50IHJlYWxpdHksIHJlYWxpemUgdGhleSBhcmUgcmVmbGVjdGlvbnMgb2YgbW9yZSBhYnN0cmFjdCBkaW1lbnNpb25zLCBhbmQgdHJ5IHRvIGZpZ3VyZSBvdXQgd2hhdCB0aG9zZSBhcmUgKGkuZS4gZmFjdG9yIGFuYWx5c2lzLCBXYWxrZXIgKDIwMTYpKS4KCgojIyMgUmVzaWR1YWxpemF0aW9uCgpMaWtlIHRoZSBuYW1lIHN1Z2dlc3RzLCB0aGlzIGludm9sdmVzIHRoZSByZXNpZHVhbHMgZnJvbSBhIGxpbmVhciBtb2RlbC4gSWYgeW91IHJlYWxseSB3YW50IHRvIGZpdCBhIG1vZGVsIGxpa2UgdGhpczoKCmBgYApuZWdhdGl2ZV9jb25jb3JkIH4gb2NjdXBhdGlvbiArIHJlc2lkZW5jZQpgYGAKClRoZW4gd2hhdCB5b3UgbmVlZCB0byBkbyBmaXJzdCBpcyBmaXQgdGhpcyBtb2RlbC4KCmBgYApyZXNpZGVuY2UgfiBvY2N1cGF0aW9uCmBgYAoKQW5kIHJlcGxhY2UgdGhlIGByZXNpZGVuY2VgIHRlcm0gaW4gdGhlIGZpcnN0IG1vZGVsIHdpdGggdGhlICpyZXNpZHVhbHMqIG9mIHRoZSBzZWNvbmQgbW9kZWwuIE5vdywgaXQgZG9lcyBtYXR0ZXIgd2hpY2ggb3JkZXIgeW91IHJlc2lkdWFsaXplIGluLiBIb3cgZG8geW91IGNob29zZT8gQWdhaW4sIHRoZXJlJ3Mgbm8gaGFyZCBhbmQgZmFzdCBhbnN3ZXIsIGJ1dCBwbGVhc2UgZG9uJ3QgdHJ5IGl0IGV2ZXJ5IHBvc3NpYmxlIHdheSBhbmQgdGhlbiBqdXN0IGNob29zZSB0aGUgbW9kZWwgdGhhdCBsb29rcyBiZXN0IQoKIyMjIEZhY3RvciBBbmFseXNpcwoKQW5vdGhlciBhcHByb2FjaCB3b3VsZCBiZSB0byBhYmFuZG9uIHRyeWluZyB0byBsb29rIGF0IGFsbCBvZiB0aGVzZSBwcmVkaWN0b3JzIGluZGVwZW5kZW50bHkgYW5kIHRvIHRyZWF0IHRoZW0gYWxsIGFzIHJvdWdoIG1lYXN1cmVzIGNvcnJlbGF0ZWQgdG8gYSBtb3JlIGFic3RyYWN0IHZhcmlhYmxlLCB1c2luZyBzb21ldGhpbmcgbGlrZSBhIGZhY3RvciBhbmFseXNpcy4gVGhpcyBpcyB3aGF0IFtXYWxrZXIgKDIwMTYpXShodHRwczovL3JlcG9zaXRvcnkudXBlbm4uZWR1L2NnaS92aWV3Y29udGVudC5jZ2k/cmVmZXJlcj0maHR0cHNyZWRpcj0xJmFydGljbGU9MTk0MCZjb250ZXh0PXB3cGwpIGRpZC4KClRoZSBiZXN0IHdheSB0byBleHBsYWluIGEgZmFjdG9yIGFuYWx5c2lzIGlzIHRvIGltYWdpbmUgc3VydmV5aW5nIGEgYnVuY2ggb2YgcGVvcGxlIG9uIHRoZSBzdHJlZXQgYW5kIGFza2luZyB0aGVtIHRoZSBmb2xsb3dpbmcgcXVlc3Rpb25zOgoKMS4gRGlkIHlvdSBoYXZlIHRoZSBoZWF0IG9uIHRoaXMgbW9ybmluZz8KMi4gQXJlIHlvdSB3ZWFyaW5nIGEgd29vbHkganVtcGVyPwozLiBBcmUgeW91IHdlYXJpbmcgYSBzY2FyZj8KNC4gQ291bGQgeW91IHNlZSB5b3VyIGJyZWF0aCB3aGVuIHlvdSBzdGVwcGVkIG91dHNpZGU/CjUuIERpZCB5b3VyIGdsYXNzZXMgZm9nIHVwIHdoZW4geW91IHdhbGtlZCBpbnNpZGU/CjYuIERpZCB5b3UgbWFrZSB5b3Vyc2VsZiBhIGJpZyBicmVha2Zhc3Q/CjcuIFdhcyB5b3VyIHN0b21hY2ggZ3Jvd2xpbmcgd2hlbiB5b3Ugd29rZSB1cD8KCgpUaGlzIGxvb2tzIGxpa2UgeW91J3ZlIGFza2VkIHBlb3BsZSA3IGRpZmZlcmVudCBxdWVzdGlvbnMsIGJ1dCByZWFsbHksIHlvdSd2ZSBvbmx5IGVmZmVjdGl2ZWx5IGFza2VkIDIgcXVlc3Rpb25zOiAKCjEuIFdhcyBpdCBjb2xkPwoyLiBXZXJlIHlvdSBodW5ncnk/CgpJZiB5b3UgdG9vayB0aGUgcmF3IGRhdGEgZnJvbSB0aGUgNyBxdWVzdGlvbiBzdXJ2ZXkgYW5kIHJhbiBpdCB0aHJvdWdoIGEgZmFjdG9yIGFuYWx5c2lzLCBpdCB3b3VsZCAoaWRlYWxseSkgdGVsbCB5b3UgdGhhdCB0aGVyZSBhcmUgb25seSAyIGVmZmVjdGl2ZSBxdWVzdGlvbnMgaW4gaXQsIGJhc2VkIG9uIGhvdyByZXNwb25zZXMgdG8gcXVlc3Rpb25zIDEgdGhydSA1IGFyZSBjb3JyZWxhdGVkIHdpdGggZWFjaCBvdGhlciwgYW5kIDYmNyBhcmUgY29ycmVsYXRlZCB3aXRoIGVhY2ggb3RoZXIuCgoKLS0tCgoKQ29sbGluZWFyaXR5IG9mdGVuIGNvbWVzIHVwIGFzIGEgcGl0ZmFsbCBvZiBvZiBlc3RpbWF0aW5nIGVmZmVjdHMuIEJ1dCBpdCdzIGFsc28gYW4gaXNzdWUgd2hlbiB0cnlpbmcgdG8gdW5kZXJzdGFuZCBvdXIgbW9kZWxzLiBBIGNvbW1vbiwgYW5ub3lpbmcsIGFuZCB1c3VhbGx5IHZhbGlkIHF1ZXN0aW9uIGlzOgoKPiBDb3VsZG4ndCB5b3VyIGVmZmVjdCBvZiBBICphY3R1YWxseSogYmUgYW4gZWZmZWN0IG9mIEI/CgpUaGUgYmVzdCB3YXkgdG8gZGVhbCB3aXRoIHRoaXMga2luZCBvZiBjb2xsaW5lYXJpdHkgaXMgdG8gdHJ5IHRvIGtpbGwgeW91ciBlZmZlY3RzLiBEb2VzIHRoZSBzYW1lIGVmZmVjdCBob2xkIGluIHRoZSBzYW1lIGRpcmVjdGlvbiBmb3IgaW1wb3J0YW50IHN1YnNldHMgb2YgdGhlIGRhdGE/IENhbiB5b3Ugc3Vic2V0IHRoZSBkYXRhIGFjY29yZGluZyB0byBCIGFuZCBmaXQgc2VwYXJhdGUgbW9kZWxzIGZvciBBPwoKCgotLS0tLS0KCgoK
